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Preface 


The  purpose  of  this  study  was  to  provide  the  Air  Force 
Materiel  Command  with  an  objective  measurement  of  system 
effectiveness  for  its  electronic  mail  system.  This 
objective  measurement  was  needed  to  ensure  that  the 
electronic  mail  system  is  meeting  the  users'  needs. 

This  research  used  a  survey  to  gather  data  on  user 
satisfaction.  The  statistical  analysis  of  the  data  gathered 
by  administering  the  survey  enabled  me  to  draw  conclusions 
about  the  effectiveness  of  the  electronic  mail  system  and 
offer  recommendations  for  improving  the  system' s 
effectiveness . 

I  received  an  enormous  amount  of  help  from  several 
individuals  throughout  this  research  process.  I  am 
sincerely  grateful  to  my  faculty  advisors,  Lt  Col  Phillip 
Miller  and  Captain  Marsha  Kwolek,  for  their  suggestions, 
guidance,  and  timely  assistance.  I  am  also  grateful  to  Mr. 
Kevin  Kampman  and  Major  Maureen  Casey  of  the  Air  Force 
Materiel  Command  for  their  help  and  information.  I  also 
thank  Dr.  Guy  Shane  for  his  assistance  in  the  data  analysis. 
Finally,  I  thank  my  wife,  Szana,  and  my  children,  Ben, 
Brittany,  and  Amanda  for  their  understanding  and  patience. 


Randall  R.  Bradford 
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Abstract 

The  purpose  of  this  study  was  to  evaluate  the 
effectiveness  of  the  electronic  mail  system  installed  at  the 
Air  Force  Materiel  Command  Headquarters.  User  satisfaction 
was  measured  as  an  indicator  of  the  system' s  effectiveness . 

In  order  to  provide  an  objective  measurement  of  system 
effectiveness,  the  following  research  questions  were 
addressed:  (1)  What  is  system  effectiveness  in  relation  to 

this  particular  electronic  mail  system,  and  how  should  it  be 
measured?  (2)  What  measurement  instruments  can  be  adapted, 
modified,  or  created  to  measure  effectiveness  as  it  is 
defined  for  this  problem?  (3)  If  a  measurement  instrument 
is  administered,  what  do  the  results  of  the  measurement 
indicate,  and  how  do  the  results  compare  to  the  Office 
Automation  staff' s  perceptions? 

User  satisfaction  was  determined  to  be  the  best 
possible  measure  of  system  effectiveness  and  it  was  measured 
by  administering  a  user  satisfaction  survey.  The  data 
gathered  from  this  survey  was  analyzed  and  that  analysis 
provided  the  basis  for  concluding  that  the  electronic  mail 
system  was  meeting  the  users'  needs,  but  that  the  system 
effectiveness  could  be  improved  by  providing  training. 
Recommendations  were  offered  to  the  Office  Automation  staff 
and  suggestions  for  further  research  were  also  given. 
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MEASURING  USER  SATISFACTION  OF  THE  ELECTRONIC  MAIL  SYSTEM  AT 
AIR  FORCE  MATERIEL  CC»4MAND  HEADQUARTERS  AS  AN  INDICATOR  OF 

THE  SYSTEM'S  EFFECTIVENESS 


i.  Introduction 


General  Issue 

The  Air  Force  Materiel  Command  (AFMC)  has  installed  a 
local  area  network  designed  to  connect  3,000  users 
throughout  its  headquarters  at  Wright-Patterson  Air  Force 
Base.  One  of  the  principal  goals  of  the  network  is  to 
provide  users  with  a  powerful  electronic  mail  system  capable 
of  meeting  the  users'  communications  needs  (Strong,  1993) . 

The  AFMC  initially  allocated  $9  million  to  fund  the 
network  project  which  began  in  May,  1991  (Strong,  1993) . 
Currently,  2,600  users  are  connected  to  the  network  through 
a  series  of  35  file  servers.  The  network  is  managed  by 
AFMC' s  Office  Automation  staff  with  TRW,  Inc.  acting  as  the 
primary  contractor. 

This  local  area  network  and  its  electronic  mail  system 
represent  a  major  investment  in  an  information  system  which 
is  considered  to  be  the  model  for  the  Air  Force.  For 
example,  this  system  has  already  been  installed  at  Air 
Mobility  Command  Headquarters  at  Scott  Air  Force  Base  and  is 
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being  looked  at  by  other  Air  Force  organizations  with  great 
interest  (Strong,  1993) . 

Specific  Problem 

Technically,  there  is  no  doiobt  that  the  system 
functions.  The  network  and  the  electronic  mail  system  are 
fully  operational  and  the  Office  Automation  staff  report 
that  all  system  design  goals  have  been  achieved  (Strong, 
1993) . 

The  problem,  however,  is  that  the  Office  Automation 
staff  needs  an  objective  measurement  of  system  effectiveness 
to  ensure  that  the  system  is  meeting  the  users'  needs. 

While  it  is  true  that  the  system  is  fully  operational,  it  is 
not  necessarily  true  that  the  users  are  completely  satisfied 
with  the  system  and  are  using  the  system  to  its  fullest 
capability. 

Definition  of  Research 

This  research  will  evaluate  the  existing  measurements 
of  effectiveness  for  information  systems  which  might  be  used 
by  AFMC's  Office  Automation  staff.  This  evaluation  will 
begin  with  an  extensive  literature  review  of  the  current 
methodologies  relating  to  information  systems,  local  area 
networks,  office  automation  projects,  and  electronic  mail 
systems . 

All  available  methodologies  will  be  evaluated  in  order 
to  determine  if  a  measurement  instrtiment  already  exists 
which  can  be  applied,  with  or  without  modifications,  to  the 
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specific  needs  of  AFMC.  This  research  will  evaluate  the 
electronic  mail  system  at  AFMC  within  the  context  of  an 
office  automation  system. 

Research  Questions 

In  order  to  provide  an  objective  measurement  of  system 
effectiveness  for  AFMC' s  electronic  mail  system,  the 
following  research  questions  must  be  answered. 

1.  What  is  system  effectiveness  in  relation  to  this 
particular  electronic  mail  system,  and  how  should  it  be 
measured? 

2.  What  measurement  instruments  can  be  adapted,  modified, 
or  created  to  measure  effectiveness  as  it  is  defined  for 
this  problem? 

3.  If  a  measurement  instaniment  is  administered,  what  do  the 
results  of  the  measurement  indicate,  and  how  do  the  results 
compare  to  the  Office  Automation  staff's  perceptions? 

Scope 

The  results  of  this  research  are  npt  generalizable  to 
areas  outside  of  the  Air  Force  Materiel  Command 
Headquarters.  This  research  is  designed  to  measure  the 
effectiveness  of  this  particular  electronic  mail  system  and 
is  not  concerned  with  other  types  of  electronic  mail 
systems . 

Although  the  measurement  results  are  limited  in  scope 
to  AFMC,  the  measurement  instrument  might  be  useful  to  other 
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organizations  which  use  a  similar  electronic  mail  system  if 
the  instryment  is  properly  validated. 

Thesis  Organization 

Chapter  II  reviews  the  literature  relating  to  this 
topic  and  addresses  the  issues  raised  in  the  first  two 
research  questions.  Chapter  III  explains  the  methodology 
used  to  answer  the  third  research  question  and  describes  the 
population  and  sample  of  interest.  Chapter  III  also 
describes  the  survey  instrument,  the  data  collection  plan, 
and  the  data  analysis  plan.  Chapter  IV  analyzes  the 
findings  of  the  survey.  Finally,  Chapter  V  lists  the 
conclusions  and  recommendations  for  further  research. 
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II .  Literature  Review 


Introduction 

This  chapter  defines  electronic  mail  and  describes  the 
features  of  the  BeyondMail  syst«n  which  is  used  at  AFMC. 

The  remainder  of  this  chapter  is  devoted  to  answering  the 
first  two  research  questions  of  this  study  through  a  review 
of  the  literature  related  to  computer-based  information 
systems  and  electronic  mail. 

Electronic  Mail 

The  essential  features  of  communication  are  the 
message,  the  sender,  the  receiver,  and  the  medium  (Stamper, 
1991:14).  Electronic  mail  is  a  computer-mediated  form  of 
communication  capable  of  delivering  messages  from  a  sender 
to  a  receiver  almost  instantaneously  without  the  use  of 
paper,  envelopes,  or  physical  delivery  (White,  1991:2).  As 
such,  electronic  mail  differs  from  regular  mail  primarily  in 
the  medium  used  (Trudell,  1984:15).  Within  an  organization, 
electronic  mail  is  typically  established  through  a  computer 
network . 

There  are  many  network  applications.  One  of  the  most 
important  applications  is  office  automation  which  uses 
electronic  mail  as  a  means  of  communication.  The  use  of 
microcomputers  combined  with  electronic  mail  and  document 
exchange  systems  has  significantly  changed  the  way  offices 
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function  (Stamper,  1991:269).  The  BeyondMail  system 
installed  at  AFMC  is  the  key  part  of  the  organization' s 
office  automation  efforts  and  offers  a  wide  variety  of 
capabilities . 

Features  of  BevondMail  Version  1.1 

BeyondMail  is  the  software  used  for  electronic  mail  at 
AFMC.  BeyondMail  is  definitely  a  state-of-the-art 
electronic  mail  program.  Some  of  the  most  significant 
features  include  messaging,  editing,  handling  options,  and 
customizing  forms  features  which  provide  the  user  with  a 
wide  variety  of  tools  for  communication  needs  and  office 
automation  services  (BeyondMail  Specification  Sheet,  1992) . 

Messaging  and  Editing.  BeyondMail  allows  the  user  to 
send,  receive,  read, . reply  to,  and  forward  messages.  The 
software  also  allows  the  user  to  convert  messages  to  drafts 
for  editing.  Once  a  message  is  received,  the  user  can 
convert  the  message  to  a  draft,  make  changes  on  the  screen, 
and  reply  directly  back  to  the  sender.  Editing  features 
include  the  ability  to  cut,  copy,  paste,  move,  and  clear 
information  (BeyondMail  Specification  Sheet,  1992) . 

Handling  Potions.  BeyondMail  software  provides  an 
encryption  capability  which  is  of  significant  interest  to 
many  users  who  send  and  receive  sensitive  information.  The 
software  also  allows  the  sender  to  set  mail  priorities  such 
as  urgent,  low,  or  regular.  Another  interesting  feature  is 
the  ability  to  request  a  return  receipt.  This  feature  could 
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be  useful  to  a  user  who  needs  to  ensure  that  a  message  is 
received  (BeyondMail  Specification  Sheet,  1992) . 

Forms.  One  feature  of  BeyondMail  software  which  could 
be  quite  useful  in  an  organization' s  efforts  to  reduce  paper 
usage  is  the  ability  to  create  electronic  versions  of 
business  forms.  The  software  will  allow  the  user  to 
customize  forms  with  the  BeyondMail  Forms  Designer 
(BeyondMail  Specification  Sheet,  1992) . 

Potential  Problem 

Even  though  BeyondMail  is  a  technologically  advanced 
electronic  mail  system,  users  who  are  dissatisfied  with  an 
electronic  mail  system  will  not  use  the  system  to  its 
fullest  capability  (Ives  and  others,  1983:786) .  The 
specific  research  problem  stated  in  Chapter  I  identified 
this  problem  and  called  for  an  objective  measurement  of 
system  effectiveness  to  ensure  that  the  system  is  actually 
meeting  the  users'  needs. 

Answering  the  First  Two  Research  Questions 

The  first  two  research  questions  can  be  answered 
through  a  review  of  previous  research  in  this  area.  There 
has  been  a  great  deal  of  research  designed  to  provide  an 
objective  measurement  of  system  effectiveness  in  relation  to 
information  systems,  office  automation  systems,  and 
electronic  mail. 

Research  Question  1.  The  first  research  question  asks 
what  system  effectiveness  is  in  relation  to  this  particular 
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electronic  mail  system  and  how  it  should  be  measured.  The 
answer  obtained  from  the  following  literature  review  is  that 
user  satisfaction  is  an  acceptable  surrogate  measurement  of 
system  effectiveness  and  it  should  be  measured  subjectively. 

Difficulties  in  Measurement.  To  begin,  it  is  generally 
not  possible  to  directly  measure  the  impact  of  an 
information  system  in  terms  of  productivity  benefits  or 
other  economic  cost  and  benefits  measures  (Raymond, 

1987:173;  Ives  and  Others,  1983:785).  In  fact,  this  author 
could  not  find  a  single  research  source  which  directly 
measured  the  impact  of  an  information  system  in  economic 
terms. 

The  difficulty  in  measuring  the  economic  cost  and 
benefits  of  a  system  or  its  effectiveness  is  largely  due  to 
the  nature  of  information  systems  themselves  and  how  they 
are  used  within  organizations.  For  example,  intangible 
costs  and  benefits  of  information  systems  are  difficult  to 
recognize  and  express  in  monetary  terms.  Also,  some  systems 
are  used  in  an  unstructured  environment  for  a  wide  variety 
of  purposes,  thus  making  the  resulting  benefits  of  the 
system  practically  impossible  to  assess.  Finally, 
information  regarding  information  system  success  may  not  be 
recorded  by  an  organization  even  when  it  would  be 
determinable,  thereby  making  the  information  unavailable  for 
research  studies  (Ives,  and  others,  1983:786). 

For  these  reasons,  researchers  have  attempted  to  find 
acceptable  measures  of  system  success  which  would  provide  an 
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organization  with  a  viable  stibstitute  for  economic  or 
productivity  measures.  Throughout  the  literature,  user 
satisfaction  has  been  widely  accepted  as  a  surrogate  measure 
for  system  success  (Bailey  and  Pearson,  1983:530;  Raymond, 
1987:173;  Ives  and  others,  1983:785;  Tan  and  Lo,  1990:203; 
Hiltz  and  Johnson,  1990:739). 

User  Satisfaction  Defined.  User  satisfaction  is 
defined  as  the  "extent  to  which  users  believe  the 
information  system  available  to  them  meets  their  information 
requirements"  (Ives  and  others,  1983:785).  As  such,  user 
satisfaction  is  a  subjective  measure  of  system  success.  In 
partial  answer  to  the  first  research  question,  therefore, 
user  satisfaction  must  be  measured  subjectively  by  asking 
the  users  what  their  satisfaction  level  is. 

Basis  for  Accepting  User  Satisfaction.  The  research  in 
this  area  began  with  the  work  of  Cyert  and  March,  who 
developed  the  original  concept  of  user  information 
satisfaction  in  their  book  entitled,  A  Behavioral  Theory  of 
the  Fiinn.  published  in  196'^.  In  their  empirical  research, 
Cyert  and  March  found  that  when  an  information  system 
successfully  meets  the  needs  of  the  users,  the  users' 
satisfaction  with  that  system  is  reinforced.  Therefore,  the 
users  will  be  more  likely  to  use  the  system  in  the  future 
and  the  satisfaction  with  the  system  will  continue  to  be 
reinforced  (Cyert  and  March,  1963:124-127).  According  to 
Bailey  and  Pearson,  another  early  researcher,  Evans,  found 
that  users  will  cease  using  a  system  completely  and  search 


for  alternative  sources  of  infoanoatlon  when  the  system  fails 
to  meet  the  users'  needs  (Bailey  and  Pearson,  1983:530) . 

The  concept  of  user  satisfaction  was  further  developed 
by  the  empirical  research  of  two  other  studies.  Swanson 
conducted  research  which  found  a  high  correlation  between  a 
user's  satisfaction  with  an  information  system  and  his  or 
her  utilization  of  its  outputs  (Swanson,  1974:184-186). 

This  research  further  developed  the  connection  between  user 
satisfaction  and  system  success. 

The  other  study  which  developed  this  connection  was 
accomplished  by  Powers  and  Dickson.  Powers  and  Dickson 
studied  10  organizations  in  a  field  study  which  evaluated 
several  criteria  of  information  system  success.  Of  all 
those  criteria  evaluated.  Powers  and  Dickson  found  user 
satisfaction  to  be  the  most  critical  criteria  for 
information  system  success.  This  conclusion  was  based  on 
the  empirical  results  of  their  research  and  their  conclusion 
that,  despite  all  other  computer  operation  considerations, 
if  an  information  system  does  not  satisfy  the  user  it  is 
intended  to  serve,  the  system  is  a  failure  (Powers  and 
Dickson,  1973:153) . 

In  the  case  of  AFMC,  user  satisfaction  is  the  specific 
determinant  of  system  success  which  the  Office  Automation 
staff  is  interested  in  measuring  (Kampman,  1993) . 

Therefore,  based  upon  the  work  of  previous  researchers,  user 
satisfaction  is  accepted  as  the  measure  of  system 
effectiveness  at  AFMC. 


Raaearch  Question  2.  The  second  research  question  asks 


what  measurement  instrtiment  can  be  adapted,  modified,  or 
created  to  measure  effectiveness  as  it  is  defined  for  this 
problem.  Based  upon  a  review  of  prior  research,  two 
previously  validated  instruments  can  be  adapted  to  measure 
user  satisfaction  at  AFMC. 

Each  instrument  is  based  upon  extensive  research  and 
has  been  tested  and  applied  in  many  settings.  The  first 
instrument,  developed  by  Tan  and  Lo,  measures  user 
satisfaction  in  the  context  of  an  office  automation  system. 
The  second  instrument,  developed  by  Hiltz  and  Johnson, 
measures  user  satisfaction  in  the  context  of  a 
computer-mediated  communication  system.  Each  instrument 
offers  a  unique  perspective  valuable  to  this  research. 

These  two  instruments  build  upon  the  work  of  Bailey  and 
Pearson.  In  1983,  Bailey  and  Pearson  evaluated  all  earlier 
attempts  to  measure  user  satisfaction  in  an  effort  to 
develop  an  effective  measurement  instrument.  Bailey  and 
Pearson  did  not  find  an  existing  instrument  which  provided 
such  a  measure.  Since  the  work  of  Bailey  and  Pearson  in 
1983,  this  author  has  not  found  any  instruments  in  the 
literature  other  than  the  ones  presented  here  which  build 
upon  Bailey  and  Pearson's  work. 

Tan  and  Lo:  User  Satisfaction  and  Office 
Automation .  Tan  and  Lo  adapted  and  validated  Bailey  and 
Pearson' s  research  instrument  to  measure  user  satisfaction 
in  the  context  of  an  office  automation  system  with 
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electronic  mail  being  a  primary  application  of  office 
autcMnation  systems  (Tan  and  Lo,  1990:203)  .  This  is 
applicable  to  AFMC  in  that  BeyondMail  is  the  primary 
application  of  the  office  autcmiation  system. 

Bailey  and  Pearson  developed  a  research  instrument 
consisting  of  39  factors  designed  to  measure  user 
satisfaction  with  computer  systems  (Bailey  and  Pearson, 
1983:530).  Tan  and  Lo  selected  33  of  the  original  39 
factors  for  their  study  of  office  autonaation  (Tan  and  Lo, 
1990:204).  Six  of  the  original  39  factors  were  removed  from 
the  research  instrument  because  those  questions  applied  to 
broader  applications  in  management  information  systems  and 
did  not  directly  apply  to  office  automation.  For  example, 
questions  concerned  with  items  such  as  charge-bacJc  for 
service  and  vendor  support  were  not  considered  relevant  to 
the  study  of  office  automation  specifically. 

Tan  and  Lo' s  research  instrument  can  be  further  refined 
to  measure  user  satisfaction  with  the  BeyondMail  system  at 
AFMC  by  eliminating  factors  which  are  not  applicable  to  AFMC 
and  the  BeyondMail  system.  Also,  the  wording  of  the 
questions  can  be  adapted  to  the  system  being  studied. 

Hiltz  and  Johnson:  User  Satisfaction  and 
Computer-Mediated  Communication  Systems.  Hiltz  and  Johnson 
accepted  the  previous  arguments  that  user  satisfaction  is  an 
acceptable  measure  of  system  effectiveness  but  concentrated 
on  electronic  mail  as  a  computer-mediated  communication 
system  (CMCS) .  Hiltz  and  Johnson  defined  electronic  mail  as 
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"message  systems  that  deliver  discrete  text  communications 
from  a  sender  to  one  or  more  recipients  via  c«nputer 
networks"  (Blitz  and  Johnson,  1990:740). 

Blitz  and  Johnson  used  the  systems  contingency  approach 
which  views  the  CMCS  as  a  "huaian-ccxaputer  system”  (Blitz  and 
Johnson,  1990:739).  This  system  is  composed  of  the  users, 
the  CMCS,  and  the  hardware  through  which  the  users  access 
the  CMCS.  Blitz  and  Johnson's  research  was  built  upon  the 
work  of  previous  studies  of  CMCS  (Blitz  and  Johnson, 
1990:739) . 

Blitz  and  Johnson  developed  a  research  instrument 
containing  14  questions  designed  to  measure  user 
satisfaction  with  a  CMCS.  The  14  questions  measure  an 
information  system  in  the  more  basic  context  of  an 
electronic  mail  system.  While  the  BeyondMail  system  at  AFMC 
is  a  significant  part  of  the  office  automation  project,  it 
can  also  be  studied  in  the  context  of  simply  being  an 
electronic  mail  system.  The  instmnnent  developed  by  Tan  and 
Lo  will  provide  a  somewhat  broader  view  of  the  BeyondMail 
system  and  Blitz  and  Johnson's  instrument  will  provide  a 
more  basic  view.  Combined,  these  instruments  should  provide 
a  useful  measure  of  the  BeyondMail  system  at  AFMC. 

Conclusions 

The  purpose  of  this  research  is  to  provide  the  Air 
Force  Materiel  Command  with  an  objective  measurement  of 
system  effectiveness  to  ensure  that  the  system  is  meeting 
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the  users'  needs.  This  literature  review  has  provided  an 
acceptable  measure  of  system  effectiveness  and  previously 
validated  research  instr\iments  which  can  be  used  to  measure 
that  effectiveness.  Chapter  III  will  provide  a  methodology 
for  applying  these  research  instruments  in  a  field  study  at 
the  Air  Force  Materiel  Command  in  order  to  answer  the  third, 
and  final  research  question. 
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III .  Methodology 


Introduction 

This  study  addresses  three  research  questions  as  stated 
in  Chapter  I.  The  research  questions  ask: 

1.  What  is  system  effectiveness  in  relation  to  this 
particular  electronic  mail  system,  and  how  should  it  be 
measured? 

2.  What  measurement  instruments  can  be  adapted,  modified, 
or  created  to  measure  effectiveness  as  it  is  defined  for 
this  problem? 

3.  If  a  measurement  instrument  is  administered,  what  do  the 
results  of  the  measurement  indicate,  and  how  do  the  results 
compare  to  the  Office  Automation  staff's  perceptions? 

The  literature  review  answers  the  first  two  research 
questions.  This  chapter  provides  the  methodology  used  to 
answer  the  third  research  question  by  describing  the 
research  design  and  the  research .instrument  used  to  measure 
user  satisfaction  as  defined  in  Chapter  II.  Furthermore, 
validation  of  the  methodology  is  discussed,  followed  by  a 
description  of  the  data  collection  plan  and  the  data 
analysis  plan. 

Research  Design 

This  research  is  designed  to  measure  user  satisfaction, 
which  is  the  user' s  subjective  measurement  of  system  success 
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(Tan  and  Lo,  1990:204) .  Therefore,  a  survey  is  appropriate 
since  the  respondents  must  provide  a  personal  assessment  of 
that  satisfaction. 

A  mail  survey  is  preferred  over  personal  and  telephone 
interviews  for  two  reasons.  First,  a  mail  survey  is 
perceived  as  more  anonymous  (Emory  and  Cooper,  1991:38) . 
Bailey  and  Pearson  state  that  the  "questionnaire  must  be 
used  in  an  atmosphere  of  user  anonymity"  (Bailey  and 
Pearson,  1983:538).  Secondly,  due  to  the  large  number  of 
surveys,  a  mail  survey  is  the  most  practical  alternative. 

Research  Instrument 

This  section  describes  the  contents  of  the 
questionnaire  and  the  rating  scale  for  the  questions.  The 
survey  pre-test  is  reviewed  and  thei  method  for  selecting 
questions  for  the  final  survey  and  assigning  specific 
descriptions  to  those  questions  is  examined. 

Contents .  The  questionnaire  contains  nine  questions  in 
the  first  section  relating  to  demographic  information. 
Section  II  of  the  survey  contains  13  questions  from  Tan  and 
Lo' s  research  which  measure  user  satisfaction  in  the  context 
of  an  office  automation  system.  Section  III  contains  13 
questions  from  Hiltz  and  Johnson' s  research  which  measure 
user  satisfaction  of  electronic  mail  as  a  computer-mediated 
communication  system.  Finally,  Section  IV  provides  the 
respondent  with  an  opportunity  to  make  comments  or 
suggestions . 
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Throughout  this  report,  the  term  section  refers  to 
Section  II  and  Section  III  of  the  questionnaire  which 
corresponds  to  Tan  and  Lo's  research  instr\iment  and  to  Hiltz 
and  Johnson's  research  instrument,  respectively.  A  copy  of 
the  questionnaire  is  attached  in  Appendix  A. 

Rating  Scale.  Questions  in  Section  II  and  Section  III 
are  rated  on  a  Likert  scale  with  values  ranging  from  1  to  7. 
This  scale  provides  a  differentiation  between  extreme  values 
such  as  those  used  in  both  sections  of  the  questionnaire. 

The  descriptions  of  the  scales,  however,  differ  between 
Section  II  and  Section  III.  All  of  the  questions  in  Section 
II  are  rated  from  extremely  dissatisfied  to  extremely 
satisfied  while  the  questions  in  Section  III  use  a  variety 
of  descriptive  terms. 

Pre-Test .  A  pre-test  of  the  questionnaire  will  be 
administered  to  a  small  sample  of  approximately  10  users 
purposefully  chosen  for  the  pre-test  based  upon  the  user' s 
expertise  and  knowledge  of  the  system.  The  results  of  this 
pre-test  will  be  analyzed  for  areas  of  improvement.  The 
respondents  will  be  asked  to  evaluate  the  construct  of  each 
question  and  provide  feedback  on  how  well  the  survey 
instrument  measures  user  satisfaction. 

Question  Selection.  According  to  Bailey  and  Pearson, 
when  using  the  instrument  in  specific  applications,  "it  is 
reasonable  to  remove  irrelevant  factors  and  redefine  the 
factors  in  situation  specific  terms"  (Bailey  and  Pearson, 
1983:538).  This  should  also  apply  to  Hiltz  and  Johnson's 
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instrument  as  well.  Therefore,  the  specific  questions 

« 

included  in  the  final  survey  were  determined  by  analyzing 
the  results  of  the  pre-test  and  by  consulting  with  key 
personnel  within  the  Office  Automation  staff. 

Final  Question  Description.  Since  the  original 
questions  used  by  both  Tan  and  Lo's  research  and  Hiltz  and 
Johnson's  research  have  broad  and  generalized  meanings,  the 
particular  questions  used  in  the  final  survey  were  made 
clearer  by  using  a  vocabulary  specific  to  AFMC. 

Validation 

This  section  examines  the  reliability  and  internal 
validity  of  both  Tan  and  Lo's  and  Hiltz  and  Johnson's 
measurement  instruments.  External  validity  is  not  addressed 
due  to  the  fact  that  this  research  is  not  generalizable  to 
areas  outside  of  AFMC. 

Tan  and  Lo;  Section  II.  Tan  and  Lo's  research  is 
based  upon  the  original  work  of  Bailey  and  Pearson  and  has 
been  extensively  validated.  Internal  validity  is  discussed 
in  terms  of  content  and  construct  validity. 

Content  Validity.  Content  validity  refers  to  the 
extent  to  which  an  instrximent  provides  adequate  coverage  of 
the  topic  under  study  (Emory  and  Cooper,  1991:180).  Content 
validity  is  basically  a  matter  of  judgment.  Given  the 
rigorous  manner  in  which  Bailey  and  Pearson,  Ives  et  al.  and 
Raymond  developed  a  comprehensive  list  of  scales,  along  with 
the  results  of  their  in-depth  analysis,  content  validity  of 


Tan  and  Lo' s  instrument  appears  to  be  more  than  adequate 
(Tan  and  Lo,  1990:205). 

Construct  Validity.  Construct  validity  relates 

the  measurement  instnnnent  to  the  theory  of  which  it  is  a 

part  (Emory  and  Cooper,  1991:182).  One  method  of  construct 

validation  excunines  the  correlations  between  item  scores  and 

total  scores.  In  the  study  completed  by  Tan  and  Lo,  each 

item  score  was  correlated  to  the  total  score: 

The  correlation  coefficient  for  the  measurement 
instrument  ranged  from  0.39  to  0.79  with  only  one 
correlation  coefficient  being  below  0.55  (Table  1) . 

All  the  correlations  were  significant  at  the  0.0001 
level.  If  it  can  be  assumed  that  the  total  scores  do 
measure  user  satisfaction,  these  results  could  indicate 
support  for  construct  validity  of  the  measurement 
instrument.  (Tan  and  Lo,  1990:205) 

The  other  construct  validation  method  involves  factor 

analysis.  The  results  of  the  factor  analysis  also  support 

the  construct  validity  of  the  instrument  (Tan  and  Lo, 

1990:205) . 

Reliability.  Reliability  refers  to  the  stability 
of  a  measurement  instrument  over  a  variety  of  conditions 
(Tan  and  Lo,  1990:205).  The  reliability  of  Tan  and  Lo's 
instrument  was  tested  using  the  internal  consistency  and 
split-halves  methods.  These  methods  measure  how  well  each 
of  the  various  instriament  questions  measures  the  same 
construct  or  idea.  Typically,  the  questions  on  the 
instrument  are  separated  into  two  groups  and  the 
correlations  between  the  two  groups  of  questions  are 
calculated.  If  the  correlations  are  high,  then  the 
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questions  are  said  to  measure  the  scune  entity  (Emory  and 
Cooper,  1991:186). 

For  Tan  and  Lo's  instnoment,  the  reliability  for  the 
internal  consistency  method  is  0.96  and  the  reliability  for 
the  split-halves  method  is  0.94.  For  the  purpose  of 
research,  a  reliaOaility  of  0.80  or  better  is  deemed  adequate 
(Tan  and  Lo,  1990:205) .  Therefore,  it  appears  that  the 
instrument  is  reliable. 

Hiltz  and  Johnson:  Section  III.  Each  of  the  14 
questions  taken  from  Hiltz  and  Johnson' s  research  has  been 
used  in  previous  research  and,  therefore,  has  received  a 
great  deal  of  validation  and  reliability  testing  prior  to 
Hiltz  and  Johnson's  efforts.  Hiltz  and  Johnson  state  that 
each  question  had  previously  demonstrated  acceptad>le 
internal  validity,  variability,  and  reliad>ility  (Hiltz  and 
Johnson,  1990:749). 

Data  Collection 

This  section  outlines  the  method  of  collecting  data. 

To  begin,  the  population  and  sample  size  are  defined.  Then, 
the  proportionate  stratified  sampling  technique  is 
described. 

Population.  The  population  consisted  of  2,608  users 
who  were  assigned  to  AFMC  and  worked  in  the  headquarters 
building.  A  list  of  these  users  was  obtained  from  the 
network  manager  and  the  survey  participants  were  selected 
from  this  list.  There  was  a  small  number  of  users  who 
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belonged  to  other  organizations  outside  of  AFMC,  amd  these 
users  were  excluded  from  this  study. 

Bach  Office  Automation  staff  member  who  was  directly 
responsible  for  the  develofzaent  and  implementation  of  the 
BeyondMail  system  received  a  questionnaire.  This  census  was 
separate  from  the  user  survey  and  its  results  allowed  a 
comparison  of  the  staff's  perceptions  to  the  users' 
responses . 

«^amr>iA  sigft.  In  order  to  obtain  a  95  'percent 
confidence  interval,  the  minimum  sample  size  needed  would 
be,  at  most:,  approximately  335  samples.  To  provide  the  best 
possible  estimator  of  the  population  mean,  this  research 
used  a  total  of  500  surveys  or  approximately  19  percent  of 
the  total  population.  The  formula  used  to  calculate  this 
sample  size  is  the  most  conservative  formula  found  by  this 
researcher.  The  formula, 

n  -  NZ^  * _ .25 

(d*  *  (N~l))  +  (Z*  *  .25) 

where : 

n  «  sample  size  required 

N  total  population  size  (2,608) 

d  *  precision  or  confidence  level  desired  (.05) 

Z  ■■  factor  for  confidence  level  (1.96) 
was  originally  developed  by  Krejcie  and  Morgan  and 
referenced  in  the  Handbook  in  Research  and  Evaluation  (Isaac 
and  Michael,  1981:192).  For  AFMC,  the  formula  estimates  the 
required  sample  size  to  be  335.  Prior  research  at  AFIT  has 


shown  that  a  return  rate  above  60  percent  £or  surveys 
conducted  within  military  organizations  is  quite  typical. 
Therefore,  a  sample  size  of  500  was  the  best  compromise 
between  the  desire  for  accuracy  and  the  desire  for 
efficiency. 

Proportionate  Stratified  Samniino.  This  study  used  a 
proportionate  stratified  sample.  According  to  Emory  and 
Cooper,  stratification  reduces  the  overall  variability  since 
the  variation  within  each  strata, ‘combined,  is  typical!  ^ 
lower  than  the  total  variation  (Emory  amd  Cooper,  1991:266). 
Also,  stratification  and  proportionate  sampling  are  almost 
always  more  efficient  statistically  when  compared  to  simple 
random  sampling  (Emory  and  Cooper,  1991:226). 

The  population  was  divided  into  19  mutually  exclusive 
subpopulations  or  strata.  These  19  strata  correspond  to  the 
command  staff  offices  and  command  directories  which  are 
connected  to  the  network.  Within  each  strata,  a  systematic 
sample  of  19  percent  of  the  users  was  taken.  Once  the 
number  of  users  to  be  surveyed  within  a  strata  was 
determined,  every  kth  element  in  the  strata  was  selected 

beginning  with  a  random  a tart  with  an  element  from  1  to  i. 

The  starting  element  was  selected  from  a  random  number 
table. 

Table  1  lists  the  19  command  staff  offices  euid  command 
directories,  along  with  their  population  size  and  sample 
size,  used  in  the  proportionate  stratified  sample.  As 


stated  earlier,  this  list  was  taken  from  the  electronic  mail 
system  and  shows  the  number  of  active  electronic  mail 
accounts  for  the  BeyondMail  system. 

TABLE  1 

COMtAND  STAFF  OFFICES  AND  COMMEND  DIRECTORIES  WITHIN  AFMC 


Strata  Population  Samuil « 


Office  of  Corporate 


Information 

400 

77 

Command  Chaplain 

9 

2 

Command  Historian 

9 

2 

Office  of  International 
Affairs 

13 

2 

Office  of  Intelligence 

40 

8 

Staff  Judge  Advocate 

35 

7 

Office  of  Public  Affairs 

38 

7 

Command  Surgeon 

55 

11 

Office  of  Security  Police 
Command  Civil  Engineer 

37 

7 

179 

34 

Test  and  Operations 

80 

15 

Personnel 

208 

40 

Engineering  and 

Technical  Management 

113 

22 

Financial  Management 
and  Comptroller 

236 

45 

Logistics 

435 

83 

Contracting 

117 

22 

Science  and  Technology 

115 

22 

Plans  and  Programs 

225 

43 

Recruirements 

264 

11 

Total 

2,608 

500 

Data  Analysis 

This  section  discusses  the  use  of  Cronbach' s  alpha  and 
the  statistical  tests  used  to  analyze  the  data.  The  results 
of  this  data  analysis  were  used  to  answer  the  third  research 
question. 
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Cronbach^s  Aloha.  Cronbach's  alpha  is  "one  of  the  most 
Important  and  pervasive  statistics  in  research  involving 
test  construction  and  use"  (Cortina,  1993:98).  The 
coefficient  alpha  is  a  measure  of  the  internal  consistency 
and  reliability  of  the  measurement  instrument.  It  measures 
the  variance  attributable  to  the  individual  items  and 
relates  that  variance  to  the  overall  measure  (Cortina, 
1993:98-100) . 

The  Statistical  Analysis  System  (SAS)  Proc  Corr  Alpha 
function  was  used  to  calculate  the  value  of  Cronbach's 
alpha,  R^,  which  is  a  number  between  0  and  1.  Specifically, 
the  value  of  is  the  percentage  of  variance  explained.  A 
value  close  to  1  means  that  a  large  percentage  of  the 
variance  is  explained,  whereas  a  value  close  to  0  means  that 
a  small  percentage  of  the  variance  is  explained. 

The  reliability  and  validity  of  this  research 
instrument  rests  largely  upon  the  previous  work  of  the  many 
authors  who  developed  and  refined  the  instruments  from  which 
it  was  formed.  Cronbach' s  alpha  does,  however,  provide  a 
measure  of  the  internal  consistency  and  relieUbility  of  this 
instrument  in  its  present,  modified  form. 

Statistical  Tests.  The  SAS  program  was  used  to  analyze 
the  data  gathered  by  the  survey.  To  begin,  the  Proc 
Univariate  FREQ  function  was  used  to  calculate  the  mean  and 
standard  deviation  for  each  question  in  Section  II  and 
Section  III  of  the  instrument.  The  FREQ  option  provided  the 
demographic  data  by  calculating  the  number  of  respondents 
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belonging  to  each  category  of  a  demographic  question 
(Schlotzhauer  and  Littell,  1991:95-99). 

Once  these  basic  tests  were  ccmupleted,  the  Proc  Corr 
function  whs  used  to  calculate  the  correlation  coefficients 
for  all  questions.  These  coefficients  were  examined  to 
determine  which  questions  were  highly  correlated  and, 
thereby,  warranted  further  investigation  and  analysis. 

Once  the  means  and  standard  deviations  were  calculated, 
the  Proc  TTest  function  was  used  to  determine  if  the  overall 
mean  satisfaction  level  for  the  users  was  different  th2ui  the 
overall  mean  satisfaction  level  for  the  OA  staff.  The  third 
research  question  asked  how  the  OA  staff's  perceptions 
compared  with  those  of  the  users.  This  test  provided  a 
measure  of  how  the  two  groups'  perceptions  compare. 

The  Proc  TTest  and  Proc  ANOVA  functions  were  used  to 
analyze  the  demographic  questions.  For  each  demographic 
question,  the  appropriate  test  was  performed  on  each 
question  in  Section  II  and  Section  III. 

The  Proc  TTest  function  was  used  to  analyze  the 
demographic  questions  which  contain  only  two  possible 
responses.  Specifically,  questions  1,  6,  and  7  contain  two 
possible  responses.  The  Proc  TTest  function  compared  the 
satisfaction  levels  (means)  for  the  two  groups  responding  to 
these  questions  (Schlotzhauer  and  Littell,  1991:224).  The 
Proc  TTest  function  tested  the  following  null  hypotheses  for 
questions  1,  6,  and  7: 
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1.  There  is  no  significant  difference  between  the  mean 
satisfaction  level  for  Mies  and  the  mean 
satisfaction  level  for  females. 

6.  There  is  no  significant  difference  between  the  mean 
satisfaction  level  of  those  users  who  have 
previously  used  other  electronic  stall  systots  and 
those  who  have  not. 

7.  There  is  no  significant  difference  between  the  mean 
satisfaction  level  for  those  who  have  received 
formal  training  and  those  who  have  not. 

The  Proc  AMOVh  function  was  used  to  analyze  the 
demographic  questions  which  contain  more  than  two  possible 
responses.  Specifically,  questions  2,  3,  4,  8,  and  9 
contain  multiple  possible  responses.  The  Proc  ANOVA 
function  compared  the  satisfaction  levels  (means)  for  each 
of  the  groups  responding  to  these  questions  (Schlotzhauer 
and  Littell,  1991:227).  Specifically,  the  Proc  ANOVA 
fxinctlon  was  used  to  test  the  following  null  hypotheses  for 
questions  2,  3,  4,  8  and  9: 

2.  There  is  no  significant  difference  among  the  mean 
satisfaction  levels  for  users  based  upon  their  age. 

3.  There  is  no  significant  difference  among  the  mean 
satisfaction  levels  for  users  based  upon  their 
education . 

4.  There  is  no  significant  difference  among  the  mean 
satisfaction  levels  for  users  based  upon  how  long 
the  user  has  used  a  computer. 

8.  There  is  no  significant  difference  among  the  mean 
satisfaction  levels  for  users  based  upon  how  long 
the  user  has  used  the  BeyondMail  system. 

9.  There  is  no  significant  difference  among  the  mean 
satisfaction  levels  for  users  based  upon  the 
designation  of  the  user  (civilian,  enlisted,  or 
officer) . 
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For  the  Proc  ANOVA  tests  which  identified  significant 
differences  among  the  mean  satisfaction  levels,  the  Scheffe 
method  of  multiple  cooqparisons  was  used  to  test  all  possible 
comparisons  for  significant  differences.  The  SAS  Proc  ANOVA 
SCHEFFE  function  was  used  to  perform  the  Scheffe 
ccaiparisons . 

In  comparison  to  the  Tukey  and  Bonferonni  methods  of 
multiple  ccmtparisons,  the  Scheffe  method  is  the  most 
appropriate  method  for  this  research  for  several  reasons. 

To  begin,  the  Scheffe  method  may  be  used  when  the  factor 
level  sample  sizes  are  not  equal  (Meter  and  others, 
1990:735-736).  The  factor  level  sample  sizes  refer  to  the 
classifications  within  the  responses  for  the  demographic 
variables.  The  factor  level  sample  sizes  for  this  research 
were  not  equal.  The  ^key  method,  on  the  other  hand,  can 
only  be  used  when  the  factor  level  sample  sizes  are  equal 
(Meter  and  others,  1990:735-736). 

Also,  the  Scheffe  method  gives  all  possible  contrasts 
among  factor  levels  (Meter  and  others,  1990:735-736). 
Furthermore,  the  Scheffe  method  is  better  than  the 
Bonferonni  method  when  the  number  of  contrasts  is  larger 
than  the  number  of  factor  levels  (Meter  and  others, 
1990:735-736).  For  this  research,  the  number  of  contrasts 
was  larger  than  the  number  of  factor  levels. 

The  results  of  the  Proc  Ttest  and  Proc  AMOVA  functions 
indicated  which  demographic  factors  provided  relevant 
information  to  enhance  the  understanding  of  how  those 
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information  to  enhance  the  understanding  of  how  those 
factors  relate  to  user  satisfaction.  The  results  of  these 
tests  were  analyzed  along  with  the  comments  made  by  the 
survey  respondents  in  order  to  answer  the  third  research 
question . 

Siinmarv 

This  chapter  described  the  research  design  and  the 
research  instrument  used  to  measure  user  satisfaction.  A 
description  of  the  data  collection  plan  and  the  data 
analysis  plan  was  also  given.  The  information  gathered  by 
analyzing  the  data  collected  with  the  research  instrtiment 
was  used  to  answer  the  third  research  question. 
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IV.  Data  Deacription  and  Analysis 


Introduction 

This  chapter  describes  the  data  collected  by 
administering  the  survey  instrxuaent  in  accordance  with  the 
methodology  provided  in  Chapter  III.  This  chapter  begins 
with  a  description  of  the  pre-test  results  which  preceded 
the  survey.  Then,  the  survey  results  are  given  in  the  form 
of  Cronbach's  alpha  coefficients,  demographic  information, 
descriptive  statistics,  correlation  coefficients,  and 
statistical  tests. 

In  several  tables  throughout  this  chapter,  questions  10 
through  35  are  referred  to  as  QIO  through  Q35,  respectively. 
For  the  convenience  of  the  reader,  the  survey  questions  are 
reproduced  here  for  easy  reference.  Questions  10  through  35 
are  rated  on  a  scale  from  1  to  7  with  a  value  of  1  being 
negative  or  unfavorable  and  a  value  of  7  being  positive  or 
favorable.  A  value  of  4  represents  a  neutral  position.  For 
a  complete  description  of  the  scales,  see  the  complete  copy 
of  the  survey  in  Appendix  A. 

Section  I..  Background 

1.  What  is  your  gender? 

1.  Male 

2 .  Female 
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2.  Nhat  is  your  age? 

1.  Less  tlian  25 

2.  26-32 

3.  33-39 

4.  40-46 

5.  47-53 

6.  54-60 

7 .  Over  60 

3.  What  is  your  highest  level  of  education? 

1.  Less  than  a  high  school  diploma 

2.  High  school  diploma  or  equivalent 

3.  Some  college  courses  but  no  degree 

4.  Associate's  degree  or  certificate 

5 .  Bachelor' s  degree 

6.  Master's  degree 

7 .  Doctoral  degree 

4.  How  long  have  you  used  a  computer? 

1.  Less  than  1.5  years 

2 .  1.5-3 

3.  3-5 

4.  5-10 

5.  10  or  more  years 

5.  What  is  your  office  symbol? 

6.  Is  this  (BeyondMail  at  AFMC)  the  first  electronic  mail 
system  you  have  used? 

1 .  Yes 

2.  No 

7.  Have  you  received  formal  training  on  using  the 
BeyondMail  system? 

1.  Yes 

2.  No 

8.  How  long  have  you  been  using  the  BeyondMail  system? 


1. 

0-3 

months 

2. 

3-6 

months 

3. 

6-9 

months 

4. 

9-12 

months 

5. 

12 

months 

9.  What  is  your  designation? 

1 .  Civilian 

2.  Enlisted 

3.  Officer 
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Section  II .  Satisfaction  with  Electronic  Mail 


How  satisfied  are  you  with. . 


10.  Relevancy  The  degree  to  which  BeyondMail  provides  the 
services  you  want  or  require. 

11.  ReliaOailitv  The  reliability  of  BeyondMail. 

12.  Completeness  The  c<»iprehensiveness  of  BeyondMail' s 
services  and  capabilities. 

13.  Vrtinni^  of  output  The  amount  of  information  conveyed  to 
you  through  BeyondMail. 

14.  Relationship  with  the  Office  Automation  (OA)  staff  The 
interaction  between  you  and  the  OA  staff. 

15.  Technical  expertise  of  the  QA  staff  The  computer 
technology  skills  and  expertise  exhibited  by  the  OA 
staff  in  the  area  of  technical  support. 

16.  Pnderstandino  of  systems  How  well  you  understand 
BeyondMail  and  its  use. 

17.  Decree  of  training  The  amount  and  quality  of 
instruction  you  received  to  develop  your  proficiency  in 
utilizing  the  BeyondMail  system. 

18.  Job  effects  The  changes  in  job  freedom  and  job 
performance  resulting  from  the  implementation  of  the 
BeyondMail  system. 

19.  Security  of  data  The  safeguarding  of  data  from 
misappropriation  or  unauthorized  alteration  or  loss. 

20.  Perceived  utility  Your  judgement  about  the  usefulness 
or  productivity  of  the  BeyondMail  system. 

21.  Convenience  of  access  The  ease  or  difficulty  with 
which  you  utilize  BeyondMail. 

22 .  Integration  of  systems  The  ability  of  BeyondMail  to 
communicate/transmit  data  between  systems  servicing 
different  functional  areas  or  users  at  other  locations. 
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Section  III .  Satisfaction  with  Electronic  Mail 


23.  Overall,  the  BeyondMail  system  is... 

(Extremely  Bad  to  Extremely  Good) 

24.  I  find  using  the  BeyondMail  system  to  be... 
(Boring  to  Stimulating) 

25.  I  find  the  language/mechanics  of  the  BeyondMail 
system. . . 

(Confusing  to  Understandable) 

26.  I  find  the  language/mechanics  of  the  BeyondMail 
system. . . 

(Unfriendly  to  Friendly) 


Please  indicate  your  reactions  to  using  BeyondMail: 


27. 

Hard  to  Learn 

to 

Easy  to  Learn 

28. 

Impersonal 

to 

Friendly 

29. 

Frustrating 

to 

Not  Frustrating 

30. 

Time  Wasting 

to 

Time  Saving 

31. 

Unproductive 

to 

Productive 

How  frequently  have  you  felt: 

(Questions  32  through  35  are  rated  from  always  to  never) 

32.  ...distracted  by  the  mechanics  of  the  BeyondMail 

system? 

33.  ...overloaded  with  information? 

34.  ...unable  to  express  your  views? 

35.  ...unable  to  get  an  impression  of  personal  contact? 
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Pre-Test  Results 


As  part  of  the  survey  development,  the  researcher 
surveyed  10  users  who  were  familiar  with  the  BeyondMail 
system.  The  10  users  were  chosen  with  the  help  of  Major 
Maureen  Casey  of  the  Corporate  Information  office.  The 
users  were  repozrted  to  be  familiar  with  the  BeyondMail 
system  in  that  they  had  been  using  the  system  since  its 
initial  implementation.  In  this  respect,  the  users  should 
have  been  as  familiar  with  the  systedi  at  AFMC  as  were  any 
other  users.  The  purpose  of  the  pre-test,  as  mentioned  in 
Chapter  III,  was  to  assess  the  construct  of  the 
questionnaire . 

The  pre-test  participants  reported  a  lack  of 
understanding  in  relation  to  the  wording  of  question  25  and 
question  26  and  suggested  that  adding  the  word  language  to 
each  question  would  make  the  questions  more  understandable. 
Based  upon  this  input,  the  word  language  was  added  to 
question  25  and  question  26. 

Other  than  this  single  discrepancy,  the  pre-test 
participants  reported  that  the  questionnaire  was  clear, 
understandable,  and  appeared  to  measure  user  satisfaction 
with  the  BeyondMail  electronic  mail  system  as  intended. 

These  results  further  demonstrated  the  construct  validity  of 
the  research  instriiment. 

Based  upon  the  results  of  the  pre-test,  the  survey 
instrument  was  finalized  and  administered  to  500  users  and 
10  Office  Automation  staff  members  at  AFMC.  Of  those 


surveyed/  260  users  and  8  Office  Automation  staff  members 
returned  useable  surveys  for  a  response  rate  of  52  percent 
for  users  and  80  percent  for  Office  Automation  staff 
members.  While  the  number  of  user  surveys  returned  was  less 
than  the  estimated  sample  size  of  335,  the  confidence  level 
associated  with  269  responses  is  between  0.05  and  0.06, 
which  is  sufficient  for  this  research. 

Cronbach^s  Alohas  and  Construct  Validity 

The  SAS  Proc  CORR  ALPHA  procedure  was  used  to  calculate 
Cronbach' s  alpha  coefficients  for  the  survey  as  a  whole  and 
for  each  question.  The  results  of  the  first  Proc  CORR  l^LPHA 
procedure  are  given  in  Table  2.  The  coefficient  for  the 
survey  as  a  whole  was  an  exceptionally  high  0.962413. 

The  Proc  CORR  ALPHA  procedure  also  calculates  what  the 
overall  Cronbach's  alpha  coefficient  for  the  survey  would  be 
if  an  individual  question  was  removed  from  the  survey.  In 
other  words,  this  is  what  the  overall  value  of  Cronbach's 
alpha  would  have  been  without  that  particular  question. 

This  calculation  is  perfomed  for  each  question  and  the 
coefficient  appears  in  Table  2  to  the  right  of  each  question 
under  the  heading  Cronbach^s  Aloha.  If  a  coefficient  which 
appears  under  this  heading  is  larger  than  the  overall 
coefficient  given  at  the  top  of  the  table,  then  that  would 
indicate  that  the  particular  question  might  be  a  bad  item 
and  should,  therefore,  be  removed  from  the  survey 
(Schlotzhauer  and  Littell,  1991:258-261) . 
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The  alpha  coefficients  for  Q17  and  Q19  were  higher  than 
0.962413,  indicating  that  the  overall  coefficient  would 
increase  if  these  two  questions  were  removed.  Therefore, 
the  Proc  CORK  ALPHA  procedure  was  performed  again  after  Q17 
and  Q19  were  removed  and  the  overall  coefficient  increased 
to  0.964140.  The  overall  coefficient  increased,  but  the 
increase  was  only  marginally  better.  In  fact,  the  increase 
was  so  small  that  Q17  and  Q19  can  be  regarded  as  good 
questions  which  could  provide  meaningful  information  and 
should,  therefore,  remain  as  part  of  the  survey. 

The  coefficients  given  in  Table  2  under  the  heading 
Correlation  With  Total  indicate  the  degree  to  which  that 
particular  question  correlates  to  the  overall  measure.  This 
value  is  inversely  proportional  to  the  value  in  the  column 
labeled  Cronbach^  s  Aloha .  This  is  due  to  the  fact  that  when 
the  question  is  highly  correlated  with  the  overall  measure, 
the  overall  value  of  Cronbach's  alpha  would  decrease  if  that 
question  was  removed  because  that  particular  question  adds 
to  the  internal  consistency  of  the  measurement  instrtiment 
rather  than  detracting  from  it. 

The  Cronbach' s  alpha  coefficient  of  0.962413  is  a 
measure  of  the  survey's  internal  consistency.  This  value 
indicates  that  96.2413  percent  of  the  variance  found  in  the 
measurements  is  explained  or  accounted  for  by  the 
instrument.  Therefore,  only  3.7587  percent  of  the  variance 
is  unexplained  or  unaccounted  for. 
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TABLE  2 


CRONBACH'S  ALPHA  COEFFICIENTS 


For  All  Variables:  0.962413 


Cronbach's  Aloha 

QIO 

0.726432 

0.960673 

Qll 

0.633258 

0.961525 

Q12 

0.733520 

0.960654 

Q13 

0.655216 

0.961271 

Q14 

0.528811 

0.962275 

Q15 

0.531298 

0.962273 

Q16 

0.612784 

0.961643 

Q17 

0.478262 

0.963243 

Q18 

0.719111 

0.960725 

Q19 

0.433458 

0.962827 

Q20 

0.789469 

0.960135 

Q21 

0.787121 

0.960083 

Q22 

0.592484 

0.962014 

Q23 

0.827542 

0.959810 

Q24 

0.760724 

0.960518 

Q25 

0.761387 

0.960332 

Q26 

0.773632 

0.960258 

Q27 

0.763309 

0.960338 

Q28 

0.824436 

0.959914 

Q29 

0.810359 

0.959841 

Q30 

0.815357 

0.959845 

Q31 

0.829597 

0.959747 

Q32 

0.732901 

0.960597 

Q33 

0.549436 

0.962166 

Q34 

0.718442 

0.960722 

Q35 

0.638328 

0.961418 

A  measure  of  this  magnitude  indicates  that  the 
survey  is  internally  consistent  and,  therefore,  provides  a 
reliable  measure  of  user  satisfaction  with  the  BeyondMail 
system  at  AFMC.  This  finding  is  consistent  with  the 
previous  research  cited  earlier  in  this  text  and  lends  a 
great  deal  of  credibility  to  the  results  which  follow.  For 
the  purpose  of  research,  a  measure  of  0.8000  or  better  is 
deemed  adequate  (Tan  and  Lo,  1990:205). 
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The  significance  of  the  Cronbach' s  alpha  coefficient 
can  hardly  be  overstated.  This  research  combined  the  work 
of  two  separate  schools  of  thought  into  one  measurement  of 
user  satisfaction.  The  two  schools  of  thought  correspond  to 
Bailey  and  Pearson' s  work  and  Hiltz  and  Johnson' s  work 
mentioned  in  Chapter  II  and  Chapter  III.  It  could  certainly 
have  been  possible  that  these  two  separate  schools  of 
thought  would  not  have  correlated  well,  or  produced  a  high 
measure  of  internal  consistency.  If  this  had  been  the  case, 
the  overall  measure  of  Cronbach' s  alpha  would  have  been  much 
lower  and  the  value  of  this  research  instrument  as  a  single 
measure  of  user  satisfaction  would  have  been  in  question. 

The  next  section  describes  the  demographic  background  of  the 
survey  respondents. 

Demographic  Data 

The  Proc  UNIVARIATE  procedure  was  used  with  the  FREQ 
option  to  calculate  the  number  of  respondents  for  each 
category  of  demographic  data.  Tables  3  through  10  list  the 
demographic  data  for  the  survey  participants.  The  tables 
correspond  to  survey  questions  1  through  4  and  6  through  9, 
respectively.  Information  concerning  the  respondent's 
office  symbol,  which  was  the  subject  of  survey  question  5, 
is  not  given  since  that  information  was  used  only  for  data 
collection  under  the  proportionate  stratified  sampling 
technique.  The  user's  office  symbol  was  not  a  variable  of 
interest  for  this  study. 
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The  item  listed  under  the  heading  DEMOGRAPHIC 


identifies  the  survey  question  or  variable  of  interest.  The 
items  under  the  heading  CATEGORY  identify  the  response  which 
categorized  the  respondents.  For  example,  in  Table  3,  the 
category  M  refers  to  the  number  of  males  responding  to  the 
survey.  Furthermore,  the  respondents  are  identified  as 
either  users  or  as  Office  Automation  (OA)  staff  members  with 
the  number  for  each  type  of  respondent  under  the  appropriate 
heading . 


TABLE  3 


DEMOGRAPHIC  DATA  -  GENDER 


DEMOGRAPHIC 

CATEGORY 

OSER 

QA  STAFF 

GENDER 

M 

164 

5 

L 

1 

TOTAL 

258 

8 

TABLE  4 

DEMOGRAPHIC  DATA 

-  AGE 

DEMOGRAPHIC 

CATEGORY 

USER 

OA  STAFF 

AGE 

Less  than  25 

6 

0 

26-32 

43 

1 

33-39 

66 

4 

40-46 

70 

2 

47-53 

50 

1 

54-60 

20 

0 

Over  60 

_3 

0 

TOTAL 

258 

8 
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TABLE  5 


DEMOGRAPHIC  DATA  -  EDUCATION 


CATEGORY 

USER 

9^  ?TMr 

EDUCATION 

Less  than  high  school 

0 

0 

High  school  ^ploma 

14 

0 

Scxae  College 

49 

1 

Associate's  degree 

28 

1 

Bachelor's  degree 

76 

4 

Master' s  degree 

84 

2 

Doctoral  decree _ 

-a 

1 

TOTAL 

259 

8 

The  demographic  background  of  the  survey  respondents 
lends  even  more  credibility  to  the  results  obtained  fr^ 
administering  the  survey.  A  review  of  the  demographic  data 
suggests  that  the  respondents  are  highly  educated  and 
possess  a  great  deal  of  computer  experience.  For  example, 
64.86  percent  of  the  respondents  have  a  bachelor's  degree  or 
higher  and  none  of  the  respondents  has  less  than  a  high 
school  diploma.  Also,  80.77  percent  of  the  respondents  have 
more  than  5  years  of  computer  experience  and  37.69  percent 
of  the  respondents  have  more  than  10  years  of  computer 
experience.  Furthermore,  83.46  percent  of  the  respondents 
were  already  familiar  with  electronic  mail  since  they 
reported  that  BeyondMail  was  not  the  first  electronic  mail 
system  that  they  had  used. 

Also  of  interest  is  the  fact  that  65  percent  of  the 
respondents  had  not  been  formally  trained  on  using  the 
BeyondMail  system  at  the  time  the  survey  was  administered. 
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♦ 

This,  conbined  with  the  fact  that  61.15  percent  of  the 
respondents  had  only  used  the  BeyondMail  system  for  less 
than  6  months,  might  account  for  some  of  the  negative 
responses  reported  in  the  next  section.  Finally,  the 
percentage  of  male  respondents  is  63.57  percent  and  the 
percentage  of  female  respondents  is  36.43  percent. 


TABLE  6 

DEMOGRAPHIC  DATA  -  CC^UTER  EXPERIENCE 


PBMPgWHIC 

CATEGORY 

USER 

OA  STAFF 

COMPUTER 

EXPERIENCE 

Less  than  1.5  years 

3 

0 

1.5-3  years 

15 

0 

3-5  years 

32 

1 

5-10  years 

112 

3 

10  or  more  years 

-2a 

1 

TOTAL 

260 

8 

TABLE  7 

DEMOGRAPHIC 

DATA  -  BEYONDMAIL 

IS  THE 

FIRST 

ELECTRONIC  MAIL  SYSTEM 

USED 

DEMOGRAPHIC 

USER 

STAFF 

BEYONDMAIL  IS 

FIRST  E-MAIL  SYSTEM 

Yes 

43 

0 

No 

217 

1 

TOTAL 

260 

8 
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TABLE  8 


DEMOGRAPHIC  DATA  -  RESPONDENTS  WHO  HAVE  RECEIVED  TRAINING 


DEMOGRAPHIC 

RECEIVED  TRAINING 

TOTAL 

g|VTE(ggftY 

Yes 

NS- 

USER 

91 

131 

260 

QA  STAFF 

0 

1 

8 

DEMOGRAPHIC 

TABLE  9 

DATA  -  TIME  USING  BEYONDMAIL 

DEMOGRAPHIC 

CATEGORY 

USER 

OA  STAFF 

TIME  USING  BEYONDMAIL 

0-3  Months 

104 

0 

3-6  Months 

55 

0 

6-9  Months 

51 

3 

9-12  Months 

30  . 

0 

12  Months  or 

more  20 

5 

TOTAL 

260 

8 

TABLE  10 

DEMOGRAPHIC  DATA  - 

DESIGNATION 

DEMOGRAPHIC 

CATEGORY 

y$ER 

DESIGNATION 

Civilian 

175 

6 

Enlisted 

24 

1 

Officer 

59 

1 

TOTAL 

258 

8 
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Descriptive  Statistics 


The  Proc  UNIVARIATE  procedure  was  used  to  calculate  the 
means  and  standard  deviations  for  each  question.  Table  11 
lists  the  means  and  standard  deviations  for  both  the  users 
and  the  Office  Automation  staff.  Missing  values  were  not 
included  in  the  calculation.  Some  values  were  rounded  to 
the  last  decimal  place  given  in  the  table. 

The  means  for  the  Office  Automation  staff  were  higher 
than  the  means  for  the  users  for  each  question  of  the 
survey.  The  procedure  discussed  in  the  next  session  was 
used  to  determine  if  these  means  are  significantly 
different. 

Overall,  given  the  impressive  technical  capabilities  of 
the  BeyondMail  software  and  the  local  area  network  used  at 
AFMC,  this  researcher  expected  that  the  ratings  given  by  the 
users  would  have  been  more  similar  to  those  given  by  the 
Office  Automation  staff.  Generally,  the  meams  for  the 
users'  satisfaction  level  are  between  4.0  and  5.0,  which 
indicates  that  their  satisfaction  level  is  from  neutral  to 
somewhat  satisfied,  with  the  exception  of  question  17. 

These  ratings  seem  to  indicate  that  the  BeyondMail  system  is 
meeting  the  users'  needs,  but  that  there  is  possibly  some 
slight  room  for  improvement. 

It  must  be  noted  that  when  analyzing  these  means  and 
standard  deviations  that  the  means  for  this  particular 
research  study  are  not  being  compared  to  an  established 
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TABLE  11 


MEANS  AND  STANDARD  DEVIATIONS 


USER  QA  STAFF 


MEAN 

STD  DEV 

MEAN 

STD  DEV 

QIO 

4.9498 

1.3842 

6.3750 

0.5175 

Qll 

4.1081 

1.5884 

5.8750 

1.1259 

Q12 

4.9112 

1.3308 

6.1250 

0.6409 

Q13 

4.8101 

1.3287 

6.7500 

0.4629 

Q14 

4.9457 

1.3570 

6.6250 

0.5175 

Q15 

5.0193 

1.3738 

6.8750 

0.3536 

Q16 

4.4115 

1.4612 

6.6250 

0.7440 

Q17 

3.4549 

1.7403 

4.8750 

1.5526 

Q18 

4.3101 

1.4103 

6.1250 

0.8345 

Q19 

4.5197 

1.1201 

5.5000 

1.5119 

Q20 

4.9154 

1.4062 

6.3750 

0.7440 

Q21 

4.9615 

1.5666 

6.7500 

0.4629 

Q22 

4.0276 

1.6903 

5.2500 

1.7525 

Q23 

4.9962 

1.4156 

6.2500 

0.8864 

Q24 

4.7344 

1.2238 

6.2500 

0.7071 

Q25 

4.9577 

1.5278 

6.6250 

0.5175 

Q26 

4.9500 

1.4444 

6.5000 

0.5345 

Q27 

4.9884 

1.4428 

6.1250 

0.6409 

Q28 

5.0000 

1.3471 

6.5000 

0.5345 

Q29 

4.6269 

1.6682 

6.6250 

0.5175 

Q30 

4.8610 

1.5084 

6.3750 

0.7440 

Q31 

4.9653 

1.4530 

6.3750 

0.7440 

Q32 

4.7441 

1.4424 

6.1250 

0.8345 

Q33 

4.9646 

1.4592 

5.6250 

1.3025 

Q34 

4.9919 

1.5492 

6.1250 

0.9910 

Q35 

4.8730 

1.4671 

6.1250 

0.9910 

benchmark  or  industry  standard.  Therefore,  interpretation 
of  the  means  relies  somewhat  upon  interpretation  by  the 
researcher  and  reader.  For  instance,  while  the  users  may 
not  be  extremely  satisfied,  they  are  at  least  generally  more 
positive  in  their  responses  than  they  are  negative.  With 
the  exception  of  question  17,  all  other  questions  had  a  mean 
greater  than  4.0000  (neutral).  While  there  may  be  some  room 
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for  improvement,  the  overall  ratings  were  at  least  greater 
than  4.00,  on  average. 

Highest  Ratings.  The  highest  rating  given  by  the  users 
is  for  question  15,  which  refers  to  the  technical  expertise 
of  the  Office  Automation  staff.  This  speaks  well  for  the 
Office  Automation  staff  and  suggests  that  the  users  perceive 
that  the  Office  Automation  staff  is  capable  of  suppoxrting 
the  local  area  network  and,  conversely,  suggests  that  any 
dissatisfaction  does  not  stem  largely  from  a  lack  of 
confidence  in  the  Office  Automation  staff's  abilities. 

The  second  and  third  highest  ratings  given  are  for 
question  28  and  question  23,  respectively.  Question  28 
rated  the  users'  reaction  to  using  BeyondMail  on  a  scale 
ranging  from  impersonal  to  friendly.  Question  23  rated  the 
users'  overall  impression  of  the  BeyondMail  system  on  a 
scale  ranging  from  extremely  bad  to  extremely  good. 

Lowest  Ratings.  The  lowest  rating  given  by  the  users 
is  the  rating  given  for  question  17,  which  measures  the 
users'  satisfaction  level  with  the  degree  of  training  the 
users  received  on  how  to  use  the  BeyondMail  system. 

Question  17  also  has  the  largest  standard  deviation,  which 
may  account  for  the  reason  that  it  appeared  to  be  a  bad 
question  when  the  first  Proc  CORK  ALPHA  procedure  was 
performed.  Furthermore,  the  lowest  rating  given  by  the 
Office  Automation  staff  is  the  rating  given  for  question  17. 

These  ratings  may  reflect  the  fact  that  65  percent  of 
the  users  and  all  of  the  Office  Automation  staff  members  who 
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responded  to  the  survey  reported  that  they  had  not  received 
any  training  on  how  to  use  the  BeyondMail  system.  It  seems 
reasonable  to  suggest  that  the  lack  of  training  evidenced  by 
the  demographic  information  presented  in  Table  8  combined 
with  the  respondents'  dissatisfaction  with  the  degree  of 
training  received  may  have  had  a  negative  impact  on  the 
overall  ratings  given.  Regardless  of  how  technologically 
advanced  an  electronic  mail  system  is,  the  system  will  not 
be  utilized  to  its  fullest  capacity  if  the  users  are  not 
trained  on  how  to  use  the  system  at  least  well  enough  to 
know  what  the  system' s  capabilities  are .  Training 
definitely  presents  itself  as  a  problem  which  deserves 
attention . 

The  second  and  third  lowest  ratings  given  are  for 
question  22  and  question  11,  respectively.  Question  22 
measured  the  users'  satisfaction  level  with  the  cd:>ility  of 
BeyondMail  to  communicate  data  between  systems  servicing 
different  functional  areas  or  users  at  other  locations. 
Question  11  measured  the  users'  satisfaction  level  with  the 
reliability  of  BeyondMail. 

Before  proceeding  with  the  analysis,  it  must  be  noted 
that  a  complete  system  failure  occurred  in  coincidence  with 
the  administration  of  the  survey.  This  event  was 
unavoidable  and  was  totally  unrelated  to  the  reliability  of 
either  the  local  area  network  or  the  BeyondMail  system.  The 
system  failure  was  due  to  an  accident  involving  a 
contractor's  technician.  The  technician  mistakenly  pushed  a 
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rack  of  equipment  against  the  wall  and  damaged  the  network 
server,  causing  a  complete  system  failure.  While  this 
failure  was  unrelated  to  the  network  or  BeyondMail,  it  may 
have  caused  some  frustration  and  anxiety  among  the  users. 
Also,  many  of  the  users  may  not  initially  have  known  the 
cause  of  the  failure  and,  thereby,  have  mistakenly 
associated  the  failure  with  the  system' s  reliability,  which 
may  help  explain  the  low  rating  for  question  11.  The  low 
rating  given  for  question  22  (the  ability  to  communicate 
between  systems  servicing  different  functional  areas  or 
users  at  other  locations),  however,  cannot  be  explained  by 
this  researcher  and  therefore  warrants  further 
investigation . 

Comments .  Section  IV  of  the  susrvey  provided  the 
respondents  with  the  opportunity  to  make  comments.  The 
comments  received  were  consistent  with  the  ratings  given. 

In  fact,  of  the  72  negative  comments  received,  the  two  most 
frequent  negative  responses  dealt  with  training  (27)  and 
reliability  (17) .  The  other  negative  comments  were  evenly 
dispersed  among  a  variety  of  items,  including  poor 
integration  with  other  systems,  abuse  of  distribution  lists, 
messages  inappropriate  for  electronic  mail,  and  the  need  for 
private  mail  groups  and  a  better  address  system. 

The  comments  suggest  that  users  are  dissatisfied  with 
the  lack  of  training  and  the  reliability  of  the  system.  In 
regards  to  training,  users  generally  stated  that  they  were 
unable  to  use  the  system  to  its  fullest  capacity  without 


proper  training.  In  fact,  many  of  the  comment  sheets  . 
returned  by  the  respondents  dealt  only  with  training.  This 
suggests  that  training  was  a  serious  enough  problem  to  cause 
the  survey  respondent  to  take  the  time  and  effort  required 
to  make  written  comments.  Training  was  the  primary  source 
of  user  dissatisfaction  reported  on  the  comment  sheets. 

Also,  in  regards  to  relicdaility,  the  users  stated  a 
dissatisfaction  with  the  amount  of  time  the  system  was 
inoperative .  Many  users  also  complained  that  the  system' s 
rate  of  message  delivery  varied  too  much.  Several  users 
commented  that  the  system  may  take  as  little  as  a  few 
minutes  to  deliver  a  message  on  some  occasions  while  taking 
more  than  a  day  on  other  occasions.  Such  a  comment  seems 
odd  to  this  researcher,  but  the  number  of  respondents  making 
the  Scune  comment  justifies  further  investigation. 

In  contrast  to  the  many  negative  comments,  there  were 
only  18  positive  comments.  The  positive  comments  ranged 
from  stating  that  the  system  is  user  friendly  to  simply 
stating  that  the  user  liked  BeyondMail.  Also,  the  positive 
comments  were  less  detailed  than  the  negative  comments. 
Overall,  the  comments  tend  to  reinforce  the  analysis  of  the 
mean  ratings  given  by  the  users. 

Pearson  Correlation  Coefficients 

To  show  the  degree  to  which  the  various  questions 
relate  to  one  another,  the  Proc  CORR  procedure  was  used  to 
calculate  the  Pearson  correlation  coefficients  (r)  for  all 


47 


possible  combinations  of  questions.  A  copy  of  all 
coefficients  is  attached  in  Appendix  D.  For  the  purpose  of 
analysis,  only  those  coefficients  greater  than  0.7500  are 
listed  in  Table  12.  Due  to  missing  values  in  the  data,  only 
223  cases  were  included  in  this  procedure.  Also,  the 
p-value  for  all  coefficients  is  0.0001.  Since  the  p-values 
are  the  same,  they  are  not  reproduced  throughout  TaUsle  12. 

The  value  of  Pearson's  correlation  coefficient  (r)  is 
given  under  the  heading  (r)  in  Table  12.  Also,  the 
questions  being  correlated  are  listed  to  the  left  of  the 
coefficient  (r)  under  the  headings  VAR  1  and  VAR  2.  For 
example,  question  20  is  correlated  to  question  23  with  a 
coefficient  of  value  0.78990.  Furthermore,  question  20  is 
correlated  to  question  31  with  a  coefficient  of  value 
0.76218. 


TABLE  12 

PEARSON  CORRELATION  COEFFICIENTS  (r) 


CORRELATED 

VARIABLES 

m. 

VAR  1 

VAR  2 

Q20 

Q23 

0.78990 

Q20 

Q31 

0.76218 

Q25 

Q26 

0.88812 

Q25 

027 

0.80324 

Q25 

Q28 

0.76298 

Q26 

Q27 

0.75206 

Q26 

Q28 

0.77244 

Q27 

Q28 

0.77984 

Q30 

Q31 

0.89758 
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These  high  correlations  illustrate  the  manner  in  which 
the  questions  are  related  to  one  another.  For  example, 
question  20  measures  the  users'  satisfaction  with 
BeyondMail's  perceived  utility  or  productivity,  and  question 
31  rates  BeyondMail  as  either  unproductive  or  productive. 

For  all  intents  and  purposes,  these  two  questions  are 
virtually  the  same  except  for  the  type  of  rating  scale  used 
and  the  exact  wording  of  the  question.  This  is  the  same 
indication  that  was  given  by  the  Cronbach' s  alpha 
coefficient  for  the  survey.  Specifically,  the  questions  all 
measure  the  same  construct  and  there  is  a  great  deal  of 
correlation  among  the  items. 

The  similarities  are  present  in  the  other  correlations 
listed  in  Table  12.  For  instance,  question  25  is  highly 
correlated  to  questions  26,  27,  and  28.  Question  25 
measures  the  users'  view  of  the  language/mechanics  of  the 
BeyondMail  system  on  a  scale  ranging  from  confusing  to 
understandable.  Similarly,  question  26  measures  the  users' 
view  of  the  language/mechanics  of  the  BeyondMail  system  on  a 
scale  ranging  from  unfriendly  to  friendly.  It  seems 
reasonable  that  if  the  user  finds  the  language /mechanics  of 
the  BeyondMail  system  to  be  understandable,  then  the  user 
would  also  find  the  language/mechanics  to  be  friendly. 

This  reasoning  is  true  for  questions  27  and  28  as  well. 
Question  27  describes  BeyondMail  as  being  either  hard  to 
learn  or  easy  to  learn  and  question  28  rates  the  users' 
reaction  to  BeyondMail  as  being  either  impersonal  or 


49 


friendly.  Again,  if  the  user  finds  the  language/mechanics 
of  BeyondMail  to  be  understandable,  then  the  user  should 
also  find  BeyondMail  easy  to  learn  and  friendly. 

This  reasoning  also  applies  to  the  correlation  between 
question  30  and  question  31.  Question  30  asks  for  the 
users'  reaction  to  using  BeyondMail  in  terms  of  being  either 
time  wasting  or  time  saving,  and  question  31  describes 
BeyondMail  as  either  unproductive  or  productive.  The 
correlation  lies  in  the  reasoning  that  if  the  system  saves 
time,  then  the  system  must  be  productive. 

Other  than  these  similarities,  it  is  noteworthy  that 
question  20  is  highly  correlated  to  question  23.  Question 
23  asks  for  the  users'  overall  satisfaction  with  BeyondMail 
and  question  20  asks  for  the  users'  satisfaction  with  the 
utility  or  productivity  of  BeyondMail.  Of  all  the  measures, 
productivity  correlates  most  closely  to  the  overall 
satisfaction  level.  This  would  seem  to  indicate  that, 
overall,  productivity,  or  utility  is  an  important  aspect  for 
users  at  AFMC. 

Users  Compared  to  the  Office  Automation  Staff 

The  Proc  TTest  procedure  was  used  to  determine  if  the 
mean  satisfaction  level  of  the  Office  Automation  staff  is 
significantly  different  from  the  mean  satisfaction  level  of 
the  users.  Table  13  lists  the  p-values  for  each  test.  The 
p— value  inr’icates  the  level  at  which  the  researcher  can 
conclude  that  a  significant  difference  between  means  exists. 
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TABLE  13 


USERS'  SATISFACTION  LEVEL  COMPARED  TO 
OFFICE  AUTOMATION  STAFF  SATISFACTION  LEVEL 


HoC  The  means  are  equal 

Variable 

P-Value 

QIO 

0.0001 

Qll 

0.0020 

Q12 

0.0007 

Q13 

0.0001 

Q14 

0.0001 

Q15 

0.0001 

Q16 

0.0001 

Q17 

0.0235 

Q18 

0.0004 

Q19 

0.0166 

Q20 

0.0006 

Q21 

0.0001 

Q22 

0.0452 

Q23 

0.0135 

Q24 

0.0006 

Q25 

0.0001 

Q26 

0.0001 

Q27 

0.0011 

Q28 

0.0001 

Q29 

0.0001 

Q30 

0.0004 

Q31 

0.0007 

Q32 

0.0076 

Q33 

0.2074 

Q34 

0.0411 

Q35 

0.0174 

This 

1  determination  is  made  by 

comparing  the  p-value  of 

the  test. 

which  is  listed  in  Table 

13  under  the  heading 

P-Value. 

to  the  reference  p-value. 

The  reference  p-value 

for  th:. 

research  is  0.10.  If  the 

p-value  of  the  test. 

which  is 

given  in  the  table,  is  less  than  the  reference 

p-value  of  0.10,  then  the  researcher  can  conclude  that  there 
is  a  significant  difference  between  the  means.  Under  this 
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condition,  the  researcher  is  rejecting  the  null  hypothesis 
that  the  means  are  equal  and  accepting  the  alternative 
hypothesis  that  the  meams  are  not  equal.  This  use  of  the 
test  p-value  given  in  a  table  and  the  reference  p-value  of 
0.10  applies  to  all  of  the  statistical  tests  presented  in 
the  remainder  of  Chapter  IV. 

For  example,  except  for  question  33,  all  of  the 
p-values  given  in  Table  13  are  less  than  the  reference 
p-value  of  0.10.  Therefore,  except  for  question  33,  the 
null  hypothesis  that  the  users'  mean  satisfaction  level  is 
equal  to  the  Office  Automation  staff's  satisfaction  level  is 
rejected  and  the  alternative  hypothesis  that  the  means  are 
not  equal  is  accepted. 

By  comparing  the  means  listed  in  Table  11,  it  is  clear 
that  the  Office  Automation  staff's  satisfaction  level  is 
significantly  higher  than  the  users'  satisfaction  level  with 
the  one  exception  previously  noted.  This  fact  suggests  that 
there  is,  indeed,  slight  room  for  improvement  since  the 
Office  Automation  staff  so  clearly  views  the  electronic  mail 
system  as  being  more  effective  than  do  the  users. 

Question  33  asked  the  respondent  how  frequently  he  or 
she  felt  overloaded  with  information.  Since  the  p— value  of 
the  test  given  in  Table  13  is  higher  than  the  reference 
value  of  0.10,  the  researcher  must  fail  to  reject  the  null 
hypothesis  that  the  mean  satisfaction  levels  are  different 
in  regard  to  question  33.  This  indicates  that  the  OA  staff 
and  the  users  do  not  differ  in  their  feeling  of  being 
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ovorloaded  with  information.  Furthermore,  given  their 
ratings  of  5.6250  and  4.9646,  respectively,  it  seems  that 
being  overloaded  with  information  is  not  a  significant 
problem  with  the  BeyondMail  system  at  AFMC. 

Statistical  Teats  for  Demographic  Questions  1.  6,  and  7 

The  Proc  TTest  procedure  was  used  to  analyze  questions 
1,  6,  and  7  since  these  questions  had  only  two  possible 
responses.  A  Proc  TTest  compares  two  separate  groups.  A 
Proc  ANOVA  procedure  is  appropriate  when  comparing  more  than 
two  groups,  which  is  the  case  for  questions  2,  3,  4,  8,  and 
9  (Schlotzhauer  and  Littell,  1991:224-227). 

These  statistical  tests  also  use  a  reference  p-value  of 
0.10.  However,  these  tests  apply  only  to  the  users  surveyed 
and  do  not  apply  to  the  Office  Automation  staff  members 
surveyed.  For  each  demographic  variable,  only  those  tests 
which  produced  a  p-value  of  0.10  or  less  are  listed. 

Tables  14,  15,  and  16  share  the  same  layout.  For  each 
table,  the  null  hypothesis  that  the  means  are  equal  is 
stated  at  the  top  of  the  tad>le.  The  question  being  examined 
is  listed  under  the  heading  VARIABLE.  For  that  question, 
the  mean  response  is  given  for  each  type  of  respondent  based 
upon  the  demographic  variable  for  that  test. 

For  example.  Table  14  lists  the  results  of  the  Proc 
TTests  for  the  gender  demographic  question.  The  possible 
categories  of  gender  include  male  and  female.  On  the 
survey,  male  corresponds  to  a  value  of  1  and  female 
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corresponds  to  a  value  of  2.  Therefore,  the  values  found 
under  the  heading  CATEGORY  identify  the  respondents  as 
either  male  or  fesiale.  The  corresponding  mean  response  or 
satisfaction  level  is  given  to  the  right  of  the  category 
under  the  heading  mean.  The  p*>value  for  the  question  is 
located  under  the  heading  P-VALUE. 

Question  1.  The  Proc  TTest  procedure  was  used  to  test 
for  a  significant  difference  in  satisfaction  level  between 
male  and  female  respondents.  The  procedure  produced  only  1 
p-value  of  0.10  or  less.  Table  14  lists  the  only  result  of 
this  procedure  with  a  p-value  of  0.10  or  less. 

TABLE  14 

PROC  TTEST  FOR  GENDER 
H,:  The  means  are  equal 

VARIABLE  CATEGORY  MEAN  P-VALUE 

Q33  1  5.0183  0.0888 

2  4.8925 


According  to  the  results  listed  in  Ted>le  14,  male  and 
female  respondents  have  a  different  mean  satisfaction  level 
for  question  33.  Question  33  asks  the  respondent  how 
frequently  he  or  she  has  felt  overloaded  with  information. 
These  results  might  indicate  that  female  users  at  AFMC  feel 
slightly  more  overloaded  with  information  encountered  in  the 
BeyondMail  system.  This  test  yields  little  real  analytical 
value  since  the  difference  in  means  is  only  0.1258.  For  all 
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practical  purposes,  a  mean  satisfaction  level  of  5.0183  is 
virtually  the  same  as  a  mean  satisfaction  level  of  4.8925. 
In  fact,  the  p-value  of  0.0888  was  only  slightly  less  than 
0.10. 

The  real  significance  of  the  Proc  TTest  procedure  for 
gender  lies  in  the  fact  that  male  and  female  respondents 
have  a  different  satisfaction  level  for  question  33  only. 

On  all  other  questions,  there  is  no  difference  in 
satisfaction  level  based  upon  the  respondent's  gender. 
Therefore,  gender  can  largely  be  ignored  as  a  significant 
factor  influencing  the  overall  satisfaction  level. 

Question  6.  The  Proc  TTest  procedure  was  used  to  test 
for  a  significant  difference  between  the  mean  satisfaction 
levels  for  those  users  who  have  previously  used  other 
electronic  mail  systems  and  those  who  have  not.  T2d>le  15 
lists  the  results  of  this  procedure  with  a  p— value  of  0.10 
or  less. 

Question  6  asked  the  respondent  if  the  AFMC  BeyondMail 
system  was  the  first  electronic  mail  system  the  respondent 
had  used.  The  possible  responses  were  yes  and  no.  A 
response  of  yes  is  indicated  in  Table  15  by  a  value  of  1 
under  the  heading  CATEGORY .  A  response  of  no  is  indicated 
by  a  value  of  2 .  For  example,  for  those  respondents  for 
whom  the  AFMC  BeyondMail  system  is  the  first  electronic  mail 
system  ever  used,  the  mean  satisfaction  level  for  question 
11  is  4.6047.  Also,  for  question  11,  the  p— value  of  the 
test  is  0.0245. 


55 


TABLE  15 


PROC  TTEST  FOR  QUESTION  6  -  FIRST  E^4AIL 


H,:  The 

means  are  equal 

VARIABLE 

CATEGORY 

MEAN 

P-VALOE 

Qll 

1 

4.6047 

0.0245 

2 

4.0093 

Q17 

1 

4.0714 

0.0117 

2 

3.3333 

Q22 

1 

4.4286 

0.0924 

2 

3.9481 

Q34 

1 

5.4146 

0.0554 

2 

4.9073 

This  test  yielded  some  interesting  results  for 
analysis.  To  begin,  question  11  dealt  with  reli2d}ility, 
question  17  dealt  with  training,  question  22  dealt  with 
integration,  and  question  34  dealt  with  the  user's  ability 
to  express  his  or  her  view  by  using  Beyondtfail.  For  each 
question,  the  mean  satisfaction  level  was  significantly 
higher  for  the  users  for  whom  this  was  the  first  electronic 
mail  system  used.  This  seems  to  indicate  that  the 
respondents  who  had  previous  electronic  mail  experience  were 
less  satisfied  with  BeyondMail  in  these  four  areas.  This 
might  indicate  that  their  expectations  were  not  completely 
met.  This  might  also  indicate  that  the  respondents  who  had 
used  another  electronic  mail  system  prior  to  using 
BeyondMail  were  better  satisfied  with  the  other  electronic 
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mail  system  in  the  areas  of  reliability,  training, 
integration,  and  the  ability  to  express  one's  view. 

Question  7.  The  Proc  TTest  procedure  was  used  to  test 
for  a  significant  difference  between  the  mean  satisfaction 
levels  for  those  users  who  have  received  formal  training  and 
those  who  have  not.  Table  16  lists  the  results  of  this 
procedure  with  a  p-value  of  0.10  or  less.  The  information 
in  this  table  follows  the  same  format  as  the  previous 
tables. 

This  test  produced  several  significant  results 
indicating  that  whether  or  not  a  user  had  received  training 
on  how  to  use  the  BeyondMail  system  influenced  his  or  her 
response  to  the  eight  questions  listed  in  Table  16.  For 
each  variable  (question)  listed,  the  mean  satisfaction  level 
for  those  users  who  had  received  training  was  significantly 
higher  than  the  mean  satisfaction  level  for  those  users  who 
had  not  received  training.  This  indicates  that  training  is 
an  important  factor  for  the  Office  Automation  staff  to 
consider  when  evaluating  the  effectiveness  of  the  BeyondMail 
system . 

The  users  who  had  received  formal  training  found  the 
BeyondMail  system  easier  to  learn  (Q27),  more  productive 
(Q31),  more  user  friendly  (Q28) ,  and  more  stimulating  (Q24) . 
than  the  users  who  had  not  received  formal  training.  These 
users  also  reported  a  better  understanding  of  the  system 
(Q16)  and  less  trouble  with  the  volume  of  output  (Q13) .  In 


fact,  the  p-value  for  question  16  was  a  highly  significant 

0.0002. 

TABLE  16 


PROC  TTEST  FOR  QUESTION  7  -  TRAINING 


H*:  The 

means  are  equal 

VARIABLE 

CATEGORY 

MEAN 

P-YALDE 

QIO 

1 

5.1667 

0.0657 

2 

4.8343 

Q13 

1 

5.0444 

0.0379 

2 

4.6845 

Q16 

1 

4.8681 

0.0002 

2 

4.1657 

Q18 

1 

4.6444 

0.0034 

2 

4.1310 

Q24 

1 

4.9545 

0.0276 

2 

4.6190 

Q27 

1 

5.2198 

0.0460 

2 

4.8623 

Q28 

1 

5.1978 

0.0680 

2 

4.8935 

Q31 

1 

5.2088 

0.0355 

2 

4.8333 

Furthermore,  the  users  who  received  training  also  found 
that  BeyondMail  provided  the  services  they  wanted  (QIO)  and 
had  a  positive  influence  on  their  job  freedom  and  job 
performance  (Q18)  to  a  higher  degree  than  the  users  who  had 
not  received  training.  These  findings  are  certainly 
reasonable  emd  emphasize  the  importance  of  training.  It 
would  seem  that  user  satisfaction  and,  therefore,  system 
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effectiveness  would  increase  if  more  users  received  formal 
training  on  how  to  use  BeyondMail. 

Statistical  Tests  for  Questions  2,  3.  4.  8.  and  9 

The  Proc  ANOVA  procedure  was  used  to  analyze  these 
questions.  Because  questions  2,  3,  4,  6,  and  9  had  more 
than  two  possible  responses,  the  Proc  ANOVA.  procedure 
follows  the  same  principles  outlined  for  the  Proc  TTests 
reported  in  the  previous  section.  The  procedure  compares 
the  mean  satisfaction  levels  for  the  users  based  upon  their 
response  to  the  demographic  question  being  analyzed. 

For  example,  the  age  demographic  variable  reported  in 
Table  17  has  seven  possible  responses  which  identify  the  age 
of  the  respondent.  The  Proc  ANOVA  compares  the  mean 
satisfaction  level  for  the  users  based  upon  their  age.  If 
the  p-value  of  the  test  is  less  than  the  reference  value  of 
0.10,  then  the  null  hypothesis  is  rejected  and  the 
researcher  concludes  than  there  is  a  significant  difference 
among  the  mean  satisfaction  levels  of  the  users  based  upon 
the  category  of  the  demographic  variable  the  user  belongs 
to.  The  Proc  ANOVA  procedure,  however,  does  not  identify 
which  categories  differ.  The  procedure  only  determines  that 
a  significant  difference  exists. 

The  SCHEFFE  option  was  used  with  the  Proc  ANOVA 
procedure  to  test  for  differences  eunong  the  categories  for 
each  question.  If  the  Proc  ANOVA  procedure  finds  that  a 
significant  difference  among  the  categories  exists,  the 
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SCHEFFE  option  can  be  used  to  determine  precisely  which 
categories  differ  (Neter  and  others ,  1990:735) . 

The  SCHEFFE  option  uses  a  default  confidence  level  of 
0.95  in  order  to  avoid  type  II  errors.  Therefore,  even 
though  the  Proc  ANOVA  procedure  finds  that  a  significant 
difference  among  categories  exists  at  the  confidence  level 
of  0.10,  the  SCHEFFE  option  may  not  be  able  to  identify 
precisely  which  categories  differ  or  by  how  much  due  to  the 
more  stringent  requirement  for  confidence  within  an  even 
smaller  range  of  values.  In  the  cases  in  this  research  in 
which  the  SCHEFFE  option  was  able  to  identify  which 
categories  differ,  the  results  are  indicated  in  the 
appropriate  table.  For  example.  Table  17  indicates  that  a 
significant  difference  exists  among  the  age  categories  for 
question  25.  The  p*-value  for  this  question  is  0.0007,  which 
is  highly  significant. 

Furthermore,  within  the  age  category,  the  SCHEFFE 
option  shows  that  a  significant  difference  exists  between 
category  2  and  category  6  respondents.  This  information  is 
found  under  the  heading  SCHEFFE  COMPARISONS.  For  question 
25,  the  information  under  this  heading  is  read  as  2  minus  6. 
The  difference  between  the  mean  satisfaction  level  for 
category  2  and  the  mean  satisfaction  level  for  category  6  is 
found  under  the  heading  DIFF  BET  MEANS. 

The  corresponding  value  for  the  difference  between 
means  for  this  question  is  1.4477.  Since  the  difference  is 
a  positive  value,  it  is  concluded  that  age  category  2 
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respondents  had  a  higher  satisfaction  level  than  age 
category  6  respondents.  By  referring  to  the  questionnaire, 
it  is  determined  that  category  2  corresponds  to  the  age 
group  26-32  and  category  6  corresponds  to  the  age  group 
54-60.  This  indicates  that  the  younger  users  find  the 
language/mechanics  of  the  BeyondMail  system  to  be  more 
understandable  than  do  the  older  users.  Such  a  finding 
suggests  that  the  older  users  have  not  adapted  to  the 
BeyondMail  system  as  t^reii  as  the  younger  users  have.  For  a 
complete  description  of  the  categories  that  follow,  refer  to 
the  questionnaire. 

Question  2.  The  Proc  ANOVA  procedure  with  the  SCHEFFE 
option  was  used  to  test  for  significant  differences  among 
the  means  for  users  based  upon  their  age.  There  were  seven 
possible  responses  to  question  2.  Table  17  lists  the 
results  of  this  procedure  with  a  p— value  of  0.10  or  less. 

Of  all  the  statistical  tests,  this  test  produced  the 
second  largest  number  of  significant  results.  The  age  of 
the  respondent,  therefore,  must  have  a  considerable  impact 
upon  the  respondent's  satisfaction  level  when  compared  to 
other  demographic  variables.  Also,  many  of  the  test 
p-values  were  highly  significant,  with  p-values  as  low  as 
0.0007.  In  fact,  only  three  tests  had  p-values  higher  than 
0.0500. 

The  SCHEFFE  option  identified  several  differences  among 
the  age  categories.  According  to  the  results  for  question 
25,  the  younger  respondents  of  age  category  2  (age  26-32) 
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TABLE  17 


PROC  ANOVA  FOR  QUESTION  2  -  AGE 


H,: 

The 

means  are  equal 

VARIABLE 

SCHEFPE  COMPARISONS 

DIFF  BET  MEANS 

Q13 

0.0149 

Q16 

0.0220 

Q20 

0.0162 

Q23 

0.0673 

Q25 

0.0007 

2-6 

1.4477 

Q26 

0.0092 

Q27 

0.0007 

2-4 

2-6 

0.9854 

1.4982 

Q28 

0.0132 

Q29 

0.0147 

Q31 

0.0105 

• 

Q32 

0.0084 

2-5 

1.0800 

Q33 

0.0095 

Q34 

0.0930 

Q35 

0.0834 

found  the 

BeyondMail 

system  to  be  less  confusing  and  more 

user  friendly  than  did  the 

older  respondents 

of  age  category 

6  (age  54-60) .  The  difference  between  these  two  means  was 
1.4477. 

Also,  for  question  27,  the  younger  respondents  of  age 
category  2  (age  26—32)  found  the  BeyondMail  system  to  be 
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easier  to  learn  than  did  the  older  respondents  of  both  age 
category  4  (age  40-46)  and  age  category  6  (age  54-60) .  The 
differences  between  these  means  were  0.9854  and  1.4982, 
respectively . 

Finally,  for  question  32,  the  younger  respondents  of 
age  category  2  (age  26-32)  were  less  distracted  by  the 
mechanics  of  the  BeyondMail  system  than  were  the  respondents 
of  age  category  5  (age  47-53) .  The  difference  between  these 
two  means  was  1.0800. 

These  results  suggest  that,  in  several  areas,  the 
younger  respondents  were  better  ed)le  to  adjust  to  the  new 
system  than  were  the  older  respondents.  In  fact,  the 
younger  respondents  had  the  higher  satisfaction  level  in  all 
comparisons  for  this  test.  For  all  practical  purposes, 
category  2  (age  26-32)  was  the  lowest  age  category  tested 
since  there  were  only  6  respondents  in  category  1  (less  than 
25)  . 

Question  3.  The  Proc  ANOVA  procedure  was  used  to  test 
for  a  significant  difference  among  the  mean  satisfaction 
levels  for  users  based  upon  their  level  of  education.  There 
were  seven  possible  responses  for  question  3.  The  SCHEFFE 
option  did  not  yield  any  significant  results  for  factor 
level  comparisons  for  question  3.  Table  18  lists  the 
results  of  this  procedure  with  a  p— value  of  0.10  or  less. 

The  results  of  this  test  indicate  that  the  respondent's 
education  level  was  not  a  highly  significant  influence  on 
the  respondent's  satisfaction  level  except  for  questions  12, 
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15,  24,  and  32.  This  might  be  partially  attributed  to  the 
fact  that  so  many  of  the  respondents  had  a  high  level  of 
education  as  indicated  in  T«d>le  5.  In  fact,  there  were  not 
even  any  respondents  to  compare  for  the  first  category  (less 
than  a  high  school  diploma)  and  there  were  only  14 
respondents  without  some  college  credit. 


TABLE  18 

PROC  ANOVA  FOR  QUESTION  3  -  EDUCATION 


The  means  are  equal 


VARIABLE 


P-VALUES 


Q12 

Q15 

Q24 

Q32 


0.0877 

0.0853 

0.0577 

0.0794 


Question  4 .  The  Proc  ANOVA  procedure  with  the  SCHEFFE 
option  was  used  to  test  for  a  significant  difference  cunong 
the  mean  satisfaction  levels  for  users  based  upon  how  long 
the  respondent  had  used  a  computer.  There  were  five 
possible  responses  to  question  4.  Table  19  lists  the 
results  of  this  procedure  with  a  p-value  of  0.10  or  less. 

This  test  found  little  difference  in  satisfaction 
levels  based  upon  the  respondent's  computer  experience.  The 
SCHEFFE  option  found  only  one  significant  difference  among 
the  categories.  Specifically,  the  respondents  with  the  most 
computer  experience  (10  or  more  years)  had  a  better 
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understanding  of  the  BeyondMail  system  than  the  respondents 
with  only  3  to  5  years  of  experience.  The  existence  of  only 
one  significant  difference  is  somewhat  surprising  in  that 
computer  experience  might  reasonably  be  expected  to  impact 
upon  a  user' s  ability  to  operate  a  new  electronic  mail 
system.  However,  except  for  question  16,  the  results 
indicate  that  computer  experience  was  generally  of  no  help. 

TABLE  19 


PROC 

ANOVA  FOR 

QUESTION  4  -  COMPUTER 

EXPERIENCE 

Ho: 

The  means  are  equal 

VARIABLE 

DIFF  BET  MEANS 

Q14 

0.0698 

Q15 

0.0494 

Q16 

0.0049 

5  ~  3 

0.9133 

Question  8.  The  Proc  ANOVA  procedure  with  the  SCHEFFE 
option  was  used  to  test  for  significant  differences  among 
the  mean  satisfaction  levels  for  users  based  upon  how  long 
the  users  had  been  using  the  BeyondMail  system.  There  were 
five  possible  responses  to  question  8.  Table  20  lists  the 
results  of  this  procedure  with  a  p-value  of  0.10  or  less. 

As  expected,  of  all  demographic  variables,  this 
varicd>le  produced  the  largest  number  of  significant  results. 
The  Proc  ANOVA  procedure  found  significant  differences  among 
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TABLE  20 


PROC  AHOVA  FOR  QUESTION  8  -  TIME  USING  BEYONDMAIL 


VARIABLE 

H,: 

P-VALUE 

The  means  are  equal 

SCHEFFE  COMPARISONS 

DIFF  BET  MEANS 

QIO 

0.0308 

Qll 

0.0092 

Q13 

0.0024 

4-1 

0.9347 

Q16 

0.0001 

5-3 

1.1863 

5-1 

1.4904 

4-1 

0.9904 

Q17 

0.0001 

5-3 

1.4300 

5-2 

1.8278 

5-1 

2.0401 

Q18 

0.0001 

5-1 

1.2712 

5-3 

1.3612 

4-1 

0.9378 

4-3 

1.0279 

Q20 

0.0640 

• 

Q21 

0.0148 

5-3 

1.3706 

Q22 

0.0922 

Q23 

0.0302 

Q27 

0.0035 

5-1 

1.0962 

Q29 

0.0045 

5-1 

1.3058 

5-3 

1.4059 

Q30 

0.0131 

Q31 

0.0202 

Q34 

0.0672 

the  mean  satisfaction  levels  for  15  of  the  survey  questions. 
Within  these  15  questions,  the  Proc  ANOVA  procedure  with  the 
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SCHEFFE  OPTION  found  15  significant  differences  among  the 
five  possible  categories  of  respondents. 

For  question  13,  the  SCHEFFE  procedure  shows  that  the 
respondents  with  more  experience  (category  4,  9-12  months) 
using  BeyondMail  are  more  satisfied  with  the  amount  of 
information  conveyed  to  them  through  the  BeyondMail  system 
than  are  the  users  with  less  experience  (category  1,  0-3 
months) . 

Also,  for  question  17,  the  SCHEFFE  procedure  shows  that 
users  with  more  experience  using  BeyondMail  (category  5,  12 
months  or  more;  category  4,  9-12  months)  have  a  better 
understanding  of  the  BeyondMail  system  in  comparison  to  the 
users  with  less  experience  (category  3,  6-9  months;  category 
1 ,  3-6  months ) . 

For  question  18,  the  SCHEFFE  procedure  shows  that  the 
users  with  more  experience  using  BeyondMail  (category  5,  12 
months  or  more)  are  more  satisfied  with  the  degree  of 
training  than  are  the  users  with  less  experience  (category 
3,  6-9  months;  category  2,  3-6  months;  category  1,  0-3 
months) .  Also,  for  the  comparison  between  users  with  12 
months  or  more  experience  using  BeyondMail  and  the  users 
with  only  0  to  3  months  experience,  the  difference  between 
means  was  a  substantial  2.0401. 

For  question  18,  the  Scheffe  procedure  also  shows  that 
the  more  experienced  users  (category  5,  12  Months  or  more; 
category  4,  9-12  months)  are  significantly  better  satisfied 
with  the  changes  in  job  performance  and  job  freedom  which 
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were  brought  about  by  the  impleaentation  of  the  BeyondMail 
system  than  are  the  users  with  less  experience  using 
BeyondMail  (category  1,  0—3  months;  categoiry  3,  6—9  months)  . 

Finally,  the  results  for  questions  21,  27,  and  29  seem 
to  also  indicate  that  the  users'  satisfaction  level  may 
increase  as  the  users  gain  more  experience  using  the 
BeyondMail  system.  For  each  of  these  questions,  the 
satisfaction  levels  for  the  more  experienced  users  were 
higher  than  the  satisfaction  levels  for  the  less  experienced 
users . 

This  pattern  is  the  same  one  which  persisted  in  each  of 
the  earlier  comparisons.  This  trend  offers  the  Office 
Automation  staff  the  hope  that  the  effectiveness  of  the 
system  will  increase  as  users  become  more  familiar  with  the 
system. 

Question  .9.  The  Proc  ANOVA  procedure  with  the  SCHEFFE 
option  was  used  to  test  for  a  significant  difference  among 
the  mean  satisfaction  levels  for  users  based  upon  their 
designation  as  either  civilian,  enlisted,  or  officer.  Table 
21  lists  the  results  of  this  procedure  with  a  p-value  of 
0.10  or  less. 

This  test  produced  only  a  few  significant  results.  The 
results  indicate  that  the  enlisted  respondents  (category  2) 
find  the  BeyondMail  system  to  be  more  user  friendly  than 
either  the  civilian  or  officer  respondents  did  (category  1, 
civilian;  category  3,  officer) .  The  lack  of  differences 
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indicates  that,  overall,  the  designation  of  the  respondent 
has  little  impact  upon  the  respondent' s  mean  satisfaction 
level . 


TABLE  21 

PKOC  ANOVA  FOR  QUESTIOH  9  -  DESIGNATION 


VARIABLE 

H,: 

P-VALOE 

The  means  are  equal 

SCHEFFE  COMPARISONS 

DIFF  BET  MEANS 

Q28 

0.0230 

2-1 

0.7712 

2-3 

0.8270 

Q34 

0.0684 

Summary 

This  chapter  described  the  data  which  was  collected  by 
administering  the  research  instrument  in  accordance  with  the 
methodology  provided  in  Chapter  III.  The  information 
collected  by  this  survey  consisted  of  Cronbach' s  alpha 
coefficients,  demographic  information,  descriptive 
statistics,  correlation  coefficients,  and  statistical  tests. 
This  information  will  be  used  to  form  the  conclusions 
presented  in  Chapter  V. 


V.  Concluaiona  Hft<‘^9»w"»endations 


Introduction 

Thia  reaearch  waa  deaigned  to  provide  an  objective 
meaaurement  of  ayatem  effectiveneaa  for  the  BeyondMail 
electronic  mail  ayatem  inatalled  at  AFMC.  Thia  effort  waa 
intended  to  determine  if  the  uaera  were  completely  aatiafied 
with  the  electronic  mail  ayatem  and,  therefore,  were  uaing 
the  ayatem  to  ita  fulleat  capacity. 

In  order  to  provide  an  objective  meaaurement  of  ayatem 
effectiveneaa  for  AFMC' a  electronic  mail  ayatem,  the 
following  reaearch  queationa  were  addreaaed: 

1.  What  ia  ayatem  effectiveneaa  in  relation  to  thia 
particular  electronic  mail  ayatem,  and  how  ahould  it  be 
measured? 

2.  What  measurement  instruments  can  be  adapted,  modified, 
or  created  to  measure  effectiveness  as  it  is  defined  for 
this  problem? 

3.  If  a  measurement  instrument  is  administered,  what  do  the 
results  of  the  measurement  indicate,  and  how  do  the  results 
compare  to  the  Office  Automation  staff's  perceptions? 

This  research  determined  that  user  satisfaction  was  the 
best  possible  measure  of  system  effectiveness  and  that  user 
satisfaction  should  be  measured  subjectively.  A  review  of 
the  literature  related  to  this  topic  revealed  two  available 
questionnaires  which  could  be  modified  and  combined  to 
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provide  a  subjective  measurement  of  user  satisfaction.  This 
research  instrument  was  administered  at  AFMC  and  the  results 
of  that  measurement  are  provided  in  Chapter  IV. 

This  chapter  draws  conclusions  based  upon  the  analysis 
of  the  research  data,  offers  recommendations  which  the 
Office  Automation  staff  might  use  to  increase  the 
effectiveness  of  the  electronic  mail  system,  suggests  topics 
for  further  research,  and  summarizes  the  research  effort. 

Conclusions 

Overall,  the  BeyondMail  electronic  mail  system  is 
meeting  the  users'  needs  at  AFMC.  Given  the  fact  that,  with 
one  exception,  the  mean  satisfaction  levels  were  more 
positive  than  they  were  neutral  or  negative,  the  users  are 
more  satisfied  than  they  are  dissatisfied.  If  there  was  any 
great  dissatisfaction  present  among  the  users  at  AFMC,  there 
would  certainly  have  been  many  mean  satisfaction  levels 
below  4.0000  (neutral).  Therefore,  it  seems  relatively  easy 
to  conclude  that  there  is  no  major  problem  with  user 
satisfaction. 

In  this  respect,  however,  the  mean  satisfaction  levels 
could  also  have  been  higher  than  they  were,  which  indicates 
that  user  satisfaction  and  system  effectiveness  could  be 
improved.  For  example,  all  mean  satisfaction  levels  were 
between  3.4549  and  5.0193.  Only  two  means  were  above 
5.0000.  The  survey  questions  were  rated  on  a  Likert  scale 
with  values  ranging  from  1  to  7.  A  value  of  7  was  the  most 
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positive  response  possible  and  a  value  of  4  was  a  neutral 
response.  In  order  for  this  researcher  to  conclude  that  the 
BeyondMail  system  is  as  effective  as  it  could  possibly  be, 
there  would  have  to  be  more  mean  satisfaction  levels  closer 
to  6.0000  or  7.0000.  The  fact  that  none  of  the  mean 
satisfaction  levels  even  came  close  to  6.0000  indicates  that 
there  is  room  for  improvement. 

Also,  it  appears  that  the  satisfaction  level  is 
steadily  increasing  as  users  gain  more  experience  with  the 
BeyondMail  system.  The  amount  of  time  an  individual  has 
used  the  BeyondMail  system  is  the  single  most  significant 
factor  affecting  user  satisfaction  level.  Those  individuals 
who  have  used  the  system  the  longest  had  significantly 
higher  satisfaction  levels  in  response  to  several  questions. 

Alternatively,  this  fact  also  suggests  that  a  learning 
curve  exists  which  initially  has  a  negative  impact  on  user 
satisfaction  and  system  effectiveness.  This  impact  is 
evidenced  by  the  fact  that  as  little  as  three  months  of 
experience  using  BeyondMail  produced  highly  significant 
differences  in  satisfaction  levels  among  users.  Also, 
several  users  made  written  comments  stating  that  BeyondMail 
seemed  to  offer  so  many  extensive  features  that  they  had 
difficulty  learning  how  to  use  the  system  and,  therefore, 
found  the  BeyondMail  system  to  not  be  as  productive  as  it 
should  be. 

The  technical  capabilities  of  the  BeyondMail  system  are 
extremely  impressive  to  this  researcher.  The  BeyondMail 
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system  at  AFMC  is  a  state-of-the-art  electronic  mail  package 
which  offers  a  comprehensive  list  of  features.  Therefore, 
it  seems  reasonable  to  expect  that  training  would  be 
essential  for  the  user  to  fully  utilize  the  BeyondMail 
system.  Unfortunately,  however,  only  35  percent  of  the 
users  had  received  training  at  the  time  the  research 
instrument  was  administered.  This  statistic  must  certainly 
be  related  to  the  fact  that  satisfaction  with  training 
received  the  lowest  overall  score  from  users.  The  mean 
satisfaction  level  of  3.4549  is  well  below  neutral  (4.0000) 
and  clearly  indicates  the  users'  c  .ssatisfaction  with 
training,  or  the  lack  thereof. 

Another  factor  of  interest  is  the  age  of  the  individual 
and  how  the  individual's  age  influences  user  satisfaction. 
The  data  showed  that  older  users  were  significantly  less 
satisfied  with  the  BeyondMail  system  in  several  areas. 
Assuming  that  older  employees  at  AFMC  generally  hold 
higher-level  positions,  it  appears  that  more  senior-level 
personnel  find  the  BeyondMail  system  to  be  less  effective 
than  do  other  users  within  the  organization.  Senior-level 
personnel  probably  have  less  time  to  learn  how  to  use  a  new 
system  such  as  BeyondMail  since  their  duties  do  not  involve 
extensive  computer  use.  Therefore,  training  is  even  more 
important  for  the  senior-level  personnel  in  order  to  reduce 
the  negative  impacts  of  the  learning  curve  associated  with  a 
new  technology  such  as  the  BeyondMail  system  at  AFMC. 
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Finally,  the  research  instrument  which  was  adapted  and 
modified  for  this  study  provided  a  useful  measure  of  user 
satisfaction  with  the  BeyondMail  system  at  AFMC.  The  data 
collected  by  administering  the  research  instrument  provided 
valuable  findings  which  were  presented  in  Chapter  IV.  While 
the  results  of  this  analysis  are  applicable  only  to  AFMC, 
the  research  instmiment  can  be  a  useful  tool  for  other 
organizations  which  have  recently  installed  an  electronic 
mail  system  or  plan  to  do  so.  Other  organizations  within 
the  Air  Force  or  within  the  Department  of  Defense  could 
easily  adapt  and  modify  this  research  instrument  to  meet 
their  specific  needs. 

Recommendations 

This  research  was  designed  to  measure  user  satisfaction 
with  the  BeyondMail  system  at  AFMC  as  an  indicator  of  the 
system's  effectiveness.  Based  upon  the  findings  presented 
in  Chapter  IV,  the  following  recommendations  are  offered  to 
the  Office  Automation  staff  at  AFMC. 

1 .  The  most  important  recommendation  that  this 
researcher  can  make  to  the  Office  Automation  staff  at  AFMC 
is  the  recommendation  to  consider  providing  comprehensive 
training  to  all  BeyondMail  users  at  AFMC.  A  special 
training  progrcun  should  be  developed  for  the  senior-level 
personnel  within  AFMC.  This  program  should  be  tailored  to 
meet  their  needs  while  considering  their  time  constraints. 
Also,  other  users  should  be  provided  with  a  choice  of  either 
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an  extensive,  detailed  training  program  for  first-time 
BeyondMall  users  or  a  less  extensive  program  for  experienced 
BeyondMall  users.  Finally,  periodic  training  could  help 
keep  users  current  and  provide  opportunities  for  the  users 
to  ask  specific  questions. 

2.  The  AFMC  Office  Automation  staff  should 
periodically  administer  the  survey  to  determine  If  user 
satisfaction  Is  Increasing  or  decreasing  and  to  measure  the 
effectiveness  of  any  training  programs  which  may  have  been 
Initiated.  The  survey  should  be  administered  once  every 
three  months  for  at  least  one  year  as  part  of  a  longitudinal 
study.  Such  a  longitudinal  study  should  provide  a  more 
complete  understanding  of  system  effectiveness. 

3.  The  survey  questionnaire  should  be  modified 
slightly  before  It  Is  administered.  To  begin,  change 
question  4  (How  long  have  you  used  a  computer?)  and  question 
8  (How  long  have  you  been  using  the  BeyondMall  system?)  so 
that  the  categories  are  mutually  exclusive.  For  example. 


categories 

1  for 

question  8  should  have  been: 

1. 

0-3 

months 

2. 

4-6 

months 

3. 

7-9 

months 

4. 

10-12 

months 

5. 

13 

months  or  more 

Also,  add  a  demographic  question  to  the  survey  which  asks 
the  user  to  Identify  his  or  her  specific  level  of 
responsibility  within  the  organization.  Question  9,  for 
Instance,  asked  the  user  to  Identify  himself  or  herself  as 
either  a  civilian,  an  enlisted  member,  or  an  officer.  A 
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♦ 

question  should  be  added  which  further  identifies  the 
respondent  as  either  0-1,  0-2,  0-7,  GS-1,  GS-9,  E-1,  E-9, 
etc . ,  so  that  senior-level  personnel  can  be  compared  to 
other  levels. 

Suggestions  for  Further  Research 

Throughout  the  course  of  this  research  project  four 
suggestions  for  further  research  have  presented  themselves 
as  important  areas  for  further  study.  These  areas  offer  the 
prospective  researcher  a  variety  of  items  to  investigate. 

1.  Administer  this  research  instrument  to  other 
organizations  with  similar  electronic  mail  systems  to 
further  refine  and  validate  the  research  instrtuaent. 

Further  refinement  and  validation  would  make  this  research 
instrvunent  a  more  useful  tool  for  organizations  throughout 
the  Air  Force  and  the  Department  of  Defense. 

2 .  Examine  other  possible  measures  of  system 
effectiveness  which  could  be  used  in  conjunction  with  this 
research  instrument  to  provide  a  more  thorough  assessment  of 
system  effectiveness. 

3.  Provide  further  justification  for,  or  argument 
against,  the  conclusion  that  user  satisfaction  is  the  best 
possible  surrogate  measurement  of  system  effectiveness  for 
electronic  mail  systems.  This  would  involve  the  development 
of  quantitative  measures  of  system  effectiveness  based  upon 
an  economic  cost  and  benefit  analysis.  Such  a  quantitative 
measure  could  also  help  an  organization  determine  if  an 
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electronic  mail  system  would  increase  an  organization's 
productivity  or  effectiveness  before  the  electronic  mail 
system  was  actually  purchased  and  installed.  User 
satisfaction  measurements,  on  the  other  hand,  are  only 
available  after  system  development  and  implementation. 

4.  Examine  the  factors  which  affect  user  satisfaction 
and  how  those  factors  influence  the  effectiveness  of  an 
electronic  mail  system.  Relevant  factors  might  include 
items  such  as  age,  education,  or  job  satisfaction.  Factor 
analysis  could  be  used  as  a  tool  to  identify  which  factors 
should  be  included  in  future  studies  of  user  satisfaction  in 
relation  to  electronic  mail  systems. 

Summary 

This  research  has  added  to  the  body  of  knowledge 
concerning  the  effectiveness  of  electronic  mail  systems. 
Specifically,  it  has  combined  the  efforts  of  several  earlier 
researchers  into  a  single  research  instrument  which  can  be 
used  by  organizations  throughout  the  military  and  the 
civilian  communities  to  measure  the  effectiveness  of  their 
electronic  mail  systems  and  to  use  that  measurement  to 
improve  the  effectiveness  of  their  systems. 

The  use  of  this  research  instrument  at  AFMC  provided 
several  significant  findings  which  the  Office  Automation 
staff  can  use  to  improve  the  BeyondMail  system.  The 
BeyondMail  system  represents  an  investment  of  several 
million  dollars  by  a  single  organization  within  the  Air 
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Force.  Likewise,  other  organizations  within  the  Air  Force 
and  the  Depairtment  of  Defense  will  certainly  be  ij^vesting 
significant  amounts  of  limited  resources  in  the  development 
and  installation  of  electronic  mail  systems.  This  research 
has  given  those  organizations  a  readily  available  evaluation 
tool  which  can  be  used  in  its  present  form  with  little  or  no 
modification. 


78 


Appendix  A:  Survey 


DEPARTMENT  OF  THE  AIR  FORCE 

MCAOQUANTIM  AM  rancs  MATIMCL  COMMAMO 
WIIMMTMATTtMON  AM  ^OnC*  SMS.  OMO 


las] 


FROM:  AFMOCS 

SUBJ:  Electtonic  Mail  Customer  Satisfaction  Suivey 
TO:  AFMC  Survey  Participants 

1.  Please  take  a  few  minutes  to  co^^)lete  the  attached  quesnoonaiie  and  return  it  by 

23  Apr  93 _ . 

2.  This  survey  measures  your  level  of  satisfiumoa  with  the  Beyond  Mail  electronic  mail  system 
and  seeks  your  comments  on  the  system's  effectiveness.  The  information  you  provide  will 
become  part  of  an  Air  Force  Institute  of  Technology  (AFTT)  research  project  awl  will  help  us  to 
determine  whether  or  not  the  electronic  mail  system  is  an  effective  tool  for  your  daily 
communications  and  office  needs. 

3.  Your  responses  will  be  combined  with  those  from  other  respondees  and  will  not  be 
attributable  to  you  personally.  Although  your  participation  is  completely  voluntary,  I  would 
apbteciate  your  help.  If  you  have  any  questions,  pleax  contact  Caftt  Randall  Bradford. 
AOT/LA,  38989,  or  Maj  Maureen  C  Casey.  613  SMSQ/CIMR.  76962. 

(l.^ickinuiKlin 
KENNETH  E.  EICKMANN 
Major  General,  USAF 
Staff  Director 


InatnietloM 


1.  Pl«aM  rMpoad  to  •aeh  qnoatioo  if  at  all  poaaibla.  If  rea  eaniiot 
aaswor  a  quMtioa  boeauaa  yoa  doa't  nadarstaiid  it  or  bocanso  yon  don’t 
want  to  anaMor  it,  than  aklp  that  qnaratioo  and  go  on  to  tho  nort 
qnoation.  Solaet  ONLY  ONI  roaponao  par  quaation. 

2.  Baeord  yonr  raaponaaa  on  tha  aaswar  ahaat  providad.  Baala  aith  tha 
^  — rfrad  ppt  uid  and  with  tha  row  narkad  035.  Baeord  your  aaswar  to 
anaation  5  on  tha  anrvay  itaalf  and  alao  on  tba  return  atwalopa.  Baeord 

mir  MBifir  Tn  ill  ithir  muBTInm  nn  tht  nigrrirr  f*^  r — *fiii 

3.  Tha  raaponaaa  will  ba  aaehina  acorad,  ao  plaasa  nark  your  answers 
with  a  Wo.  2  oaneil  and  blacken  tha  appropriate  circle  eonplataly. 

Plaasa  erase  any  stray  narks  and  do  not  fold  tha  aaswar  sheet. 

4.  Tha  last  section  of  tho  quastionaaira  provides  yon  with  tha 
opportunity  to  sake  eoanents  or  suggastiona.  You  say  consent  on  any 
aspect  of  tha  BayoadNail  systan  or  tho  surrey.  Your  coonents  are 
walconad  and  will  not  ba  attributed  to  you  personally  in  any  aannar. 

5.  When  you  have  coaplatad  tha  surrey,  plaasa  place  tha  quastionaaira 
aaswar  sheet  in  tha  aaralopa  provided.  Nark  yonr  office  synbol  on 

the  outside  of  tha  envelope  and  seal  it.  Batum  tba  survey  to: 

Capt  Bradford 
e/o  Naj  Casey 
ciia 
^t  IBQ 


MI  YOU  FOB  PABTICIPATIWQ! 


4 
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aifi&ifia  !•  ftMiktrayad 


Nta«t  is  your  gssdsr? 

1.  Msls 

2.  PSMlS 


Whst  is  your  sfs? 

1.  Loss  thus  2S 

2.  26>32 

3.  33-39 

4.  40-46 

5.  47-53 

6.  54-60 

7.  Ovsr  60 


Hhst  is  your  highest  Isvsl  of  sduestioo? 

1.  Less  thsn  s  high  school  diploss 

2.  High  school  diploas  or  squirslsst 

3.  SoM  collsgs  courses  but  no  degree 

4.  Associste's  degree  or  certificate 

5.  Bachelor's  degree 

6.  Master’s  degree 

7.  Doctoral  degree 


How  long  have  you  used  a  coaputer? 

1.  Less  than  1.5  years 

2.  1.5  -  3 

3.  3-5 

4.  5-10 

5.  10  or  sore  years 


What  is  your  office  symbol?  (Be  Specific) 
BeCOrd  rour  —««— •  Aim  «■  th. 


la  this  (BayoadNail  at  AfNC)  tha  flrat  alaetroaic  mail  ayataa  rou 

hava  uaad? 

1.  Yaa 

2.  No 


Hava  rou  racairad  foraal  traiaiaf  oa  uaiaf  tlia  Bayondllail  arataa? 

1.  Tas 

2.  Ho 


Boa  long  hara  yoo  baan  aaiag  tba  BaroadNail  Syataa? 

1.  0-3  aoBttaa 

2.  3-6  aontfaa 

3.  6-9  aoatha 

4.  9-12  aootha 

9.  12  ■OBtha  or  aora 


What  ia  yoor  daaigaatioB? 

1.  Civiliaa 

2.  bliatad 

3.  Officar 


II*  SatiMfactloB  with  Elactronie  Mail 


This  ssctios  asks  70U  to  svsiusto  jrour  lorsl  of 
amtisfsction/dissstisfsctioo  with  tho  BoyondNsil  sloctroaie  soil  systaa. 

How  sstlsfiod  are  you  with... 

10.  HolorsncT  Tho  dotroo  to  which  teyondNoil  provides  tho  sorvices  you 
wont  or  roquiro. 


Extreooiy 

Oiaastiafiod 

Neutral 

Extreooiy 

Satisfied 

Iztroaoly 

Oissotisfiod 

Neutral 

g. 

Bxtroaoly 

Satisfied 

CooDloteness  Tho  cooDrehoBoivoBooo  of  BovondMoi 1 ’a  Bapvlr>aa 
cspobiiitios. 

ExtreooLy 

Oissotisfiod 

Neutral 

V  I 

Extreooiy 

Satisfied 

Voluoo  of  output  Tho  oww 
BtyondMail. 

1-  _ 2-  *1 

nt  of  inforaatioo  convoyed  to  you  throuch 

J  C  £  9 

Extreooiy 

Dissatisfied 

Neutral 

Extreooiy 

Satisfied 
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14.  B^lmtionahip  with  thm  Qffle*  AutoMtioB  (QA>  ataff  Th*  inUractioa 
b«tN«€n  foa  and  th*  OA  staff. 


1 - 2 - 

Extrsasly 

Hautral 

Extrsasly 

Oissatisfiad 

Satisfied 

15.  Tachnleal  a^artlaa  of  tha  QA  atpff  Tha  coapntar  tachnolocy  skills 
and  axpsrtisa  axhibitad  by  tbs  OA  staff  in  tbs  arsa  of  tachniesl 
support. 


1 - 2 - 3 - 4 - S - • - 7 

Extrsasly  Rsutral  Ixtrsssly 

Oissatisfiad  Satisfisd 


16.  Ondsrstandlns  of  systsss  How  wall  you  undarstand  BayondNail  and 
its  usa. 


1 - 2 - 3 - 4 - 5- - 6 - 7 

Ixtraaaly  Hautral  Extrsasly 

Oissatisfiad  Satisfisd 


17.  Desraa  of  trainins  Tha  aaount  and  quality  of  instruction  yon 

racairad  to  daralop  your  profieiancy  in  utilixinf  the  BayondNail 

systaa. 


1 - 2 - 3 

Ixtraaaly 

Oissatisfiad 


- 4 - 

Hautral 


Ixtraaaly 

Satisfied 


18.  Job  affects  Tha  changas  in  Job  fraadoa  and  Job  parfomanca 
rasnltinf  froa  tha  iaplaaentation  of  tha  BayondNail  systaa. 


Extrsasly  Hautral 

Oissatisfiad 


Extreaely 

Satisfied 
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19.  Security  of  datm  The  safeguurdinf  of  data  froa  alaappropriation  or 
unauthorised  alteration  or  loaa. 


Extreaely  Meutral  Extreaely 

Oisaatiafied  Satiafied 


20.  Perceiyed  utility  Your  judeeaant  about  the  uaefulneaa  or 
productiyity  of  the  BeyondNall  ayataa. 


1 - 2 - 3 - 4 - 5 - 6 - 7 

Extreaely  Neutral  Bxtreaely 

Oiaaatiafied  Satiafied 


21.  Conyenience  of  acceaa  The  eaae  or  difficulty  with  which  yon 
utilise  BeyondNall. 


Extreaely 

Dissatisfied 


Neutral 


Bxtreaely 

Satisfied 


22.  Integration  of  ayateaa  The  ability  of  BeyondNail  to 

coaaunicate/transait  data  between  systeas  seryicing  different 
functional  areas  or  users  at  other  locations. 


Extreaely  Neutral 

Dissatisfied 


Extreaely 

Satisfied 
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atsUsa  HI* 


TIm  folloMinf  qnMtieaa  uk  70a  to  ovolaoto  jour  lovol  of 
Mtisfaetlon/dioMtisfaetioa  with  tho  BotomMoII  oLoetroBie  oail  systoa 
by  —iiiw  B  diffof— t  twao  of  obbIo. 


23.  Ovorallt  tho  BoyootMail  systM  is... 


Extrwooly 

Bad 


Noutral 


Extroaaly 

Good 


24.  I  find  osiac  tho  BoyoodNsil  systsa  to  bo. 


1 - 2- 

Boring 


Botttrsl 


StiaolatiBf 


2S.  I  find  tho  languago/oochaaics  of  tho  BoyoDdNail  systoa. .. 


1 - 2 - 3- 

CoafosiBg 


Rontrml 


UBdorstaadsblo 


26.  I  find  tho  laaguago/aochanics  of  tho  BoyoncBfsil  systoa... 


1 - 2- 

Unfriondly 


Neutral 


Friendly 


♦ 


PleaM  indicate  your  reactioaa  to  using  BeyondNail: 


27. 


Hard  to  Learn 


2 — 


•3- 


Neutral 


-7 

Saar  to 


Learn 


28. 


1 — 
lapersonal 


Neutral 


Friendly 


29. 


Frustrating 


3- 


Neutral 


.  Not  Frustrating 


30. 


Tise  Wasting 


-3 - 4 - 5 - 

Neutral 


Tise  Saving 


31. 


1 - 2 - 

Unproductive 


Neutral 


Productive 
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How  froqaoatly  hare  you  folt 


.distraetod  by  tha  aachanies  of  tha  BayondNail  'ayataa? 
Always  Soaatiaaa  Narar 


.overloaded  with  inforaation? 

1 - 2 - 3 - 4 - 5 - 6 - 7 

Always  Soaatiaas  Never 


.gnsbla  to  express  yoar  views? 

Always  Soaatiaas  Never 


.jBabla  to  gat  aa  iaprassios  of  personal  contact? 


Always 


Soaatiaas 


Never 


S«etlQB  11.  CoM«nta 


Pleaa*  ua«  thia  aeetioo  to  sako  any  coMMnts  or  auggeatlons  which 
■ay  holp  tho  Air  Foreo  Natoriol  Co— and  provide  you  with  the  b— t 
electronic  — il  ayat—  p— alble.  Any  coa— nta  you  aake  will  be  atrictly 
confidential. 


Thank  you! 


« 


00003738  1S3S  2211S4S4SSSSS45<5SS5SSSS6S666€ 

00003738 

00001300  1361  2223666666S5446666665666666566 
00001300 

00003487  1464  22136S666763£56S66666S6666635S 
00003487 

00001249  1454  211132433332443323334342543535 
00001249 

00001337  2525  214153554353335565555555534433 
00001337 

00001268  2434  223133646643446543535364434463 
00001268 

00001209  2354  213153556566444645544555534536 
00001209 

00001291  1255  222311775374147777477747467777 
00001291 

00001198  2334  211143955643944546559469354333 
00001198 

00001336  2435  213169666656565956667766665665 
00001336 

00001292  1365  213133534352466635633667666376 
00001232 

00001332  2434  222132347342434344333444444344 
00001352 

00003703  2334  224141435962645515677777777777 
00003703 

00001342  2434  212155353654556656955336656334 
00001342 

00003331  2224  214176777776477777767776777776 
00003331 

00003730  1563  213163664454446746655666664656 
00003730 

00001293  1564  223163336653646546676766665566 
00001253 

00001257  2469  12313333444  343235544443944354 
00001257 

00003719  1464  113366344455325595355555596367 
00003719 

00001289  2233  214131393355629337433643774576 
00001289 

00003707  1565  223162659593436636655555566566 
00003707 

00001286  1263  224363675464636765576553956666 
00001286 

00001363  1533  212164664455345636356766446665 
00001363 

00003382  1344  221292644423355434576454444474 
00003382 

00001295  1453  213151445455245312343444344335 
00001235 

00001285  1753  221153536633276345363555245735 
00001285 

00003483  2353  221143433421344434355433344334 
00003483 

00003366  2324  222133345544444424444443445644 
00003366 

00001350  2434  213143455422344944433343443443 
00001390 

00003420  1444  221155964451646767955665666666 
00003420 

00003422  1562  222173764461646757766666666666 
00003422 

00003425  15  5  2211422444524  4423433444444565  > 

00003423 

00001205  1254  113365564565566656566656664635 
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0000120S 

00001385  1244  2232SS5S66445666564S5556S55655 
00001385 

00001208  1664  212152454452445555444444544444 
00001208 

00001266  1365  222154665562546655566565665747 
00001266 

00001364  2355  213344454432445325544344554445 
00001364 

00001216  1375  223354354454545515566666666666 
00001216 

00003445  2324  211174664456456644644444444444 
00003445 

00003469  1472  111345544465424545466656445666 
00003469 

00003404  1675  214312234455342122422222242331 
00003404 

00003317  1365  221152533632246634455444554656 
00003317 

00003322  1134  115263645567353724466765446565 
00003322 

00003724  1365  1223553566214553  5555333555642 
00003724 

00003718  1352  123154  555334455554555345565 
00003718 

00001382  2665  113165775463566656555566556375 
00001382 

00003349  1565  213161655645557336655652554445 
00003349 

00001374  1465  21316566666556555545555555555 
00001374 

00003321  2344  112274527742564556555555225265 
00003321 

00007993  2254  125177777766777777777777777777 
00007993 

00003377  65  222  76667764445745477777547771 

00003377 

00003344  1545  223166566664566656666555676777 
00003344 

00001334  1664  213155556632556335433355554534 
00001334 

00001357  2224  211166665666666767666777777776 
00001357 

00001269  1564  221151544452242512445443435555 
00001269 

00003733  1263  221333334411143232223331333523 
00003733 

00003435  1562  223344644421465555323454554444 
00003435 

00008013  2654  212155657755546556555555666666 
00008013 

00003272  2  55  221154437754344435367453242466 
00003272 

00003448  2544  124133455555444444333552434433 
00003448 

00003287  1354  222176777755646777677777775667 
00003287 

00003267  2254  2111563567643347  5576777475576 
00003267 

00003353  1435  124276577764767776657777777477 
00003353 

00001345  2354  213155463466  64425576555564444 
00001345 

00001292  1564  221156444421335326545452543524 
00001292 

00003358  1144  222122353451462513377656447744 
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00003358 
00003706  1374 
00003706 
00003396  1363 
00003396 
00003264  2564 
00003264 
00001229  1265 
00001229 
00001234  2134 
00001234 
00003492  1254 
00003492 
00001251  1555 
00001251 
00003726  1443 
00003726 
00001248  1454 
00001248 
00001369  2434 
00001369 
00003709  2345 
00003709 
00001282  1562 
00001282 
00001303  1465 
00001303 
00003714  1345 
00003714 
00001195  1465 
00001195 
00001304  1343 
00001304 
00003465  1654 
00003465 
00001232  2233 
00001232 
00001299  1234 
00001299 
00003717  2464 
00003717 
00003400  1554 
00003400 
00001324  1464 
00001324 
00003443  1262 
00003443 
00001344  2365 
00001344 
00003749  2234 
00003749 
00001242  1565 
00001242 
00001333  1365 
00001333 
00003325  1565 
00003325 
00007997  1465 
00007997 
00003708  2353 
00003708 
00003407  1565 
00003407 
00001365  2655 
00001365 
00001202  1655 


213333545632245343332353552442 

221163766764667767776667675567 

1211434333314444433252322  5434 

221363526621647747766776677767 

213144445566566655577777554  666 

22114444445  4444444447774477 

223165644432233444455434342355 

221233445522446365434353543534 

221146344431254745544454454544 

225166664655546767577777776666 

224174665465666646577773664477 

2213333676345653433363656655 

221134344454343343334443333333 

221244444421445543444354453444 

221133443522443444455534353655 

221266666655566667666766667777 

221134464431145366221423352444 

123266766664546647777776676675 

122253564454453566566666667676 

222154653443646646566555534635 

224133443531445234433442241422 

221334345532445545655555444556 

112376667756576677676766667666 

1143423555312  3312233343343134 

2211633643434466766776676677 

213165665655646545677676666676 

225353354454556635555555664466 

224166666764457626555777776777 

211363556655445535555666556666 

224163664334645326555455654445 

212164646165545666666556666555 

215144454455465445545343333232 

214153455675445765444755557455 
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00001202 

00002042  2364  225166667774747777755677776767 
00002042 

00003342  2452  222154453434454535444343453254 
00003342 

00003432  1364  211336563656443755576666565476 
00003432 

00003371  2333  212134357764553635666776664554 
00003371 

00003409  1473  112333334444545535555445434534 
00003409 

00003292  1434  211163635364435524566653554554 
00003292 

00003296  1334  221266635754576643444454444534 
00003296 

00003459  2534  211133346664444414374744343423 
00003439 

00001231  1454  113166654336766556376656776577 
00001231 

00001366  2634  124144764434646566363555663255 
00001366 

00003731  2344  212123455532545525444555453554 
00003731 

00003370  2633  211135653121646576765563775553 
00003370 

00003315  2254  221166736641545526565663666777 
00003313 

00003442  1564  124377776635657777777777776777 
00003442 

00001335  1363  211166737755465645367666666666 
00001355 

00003369  2133  221143444341434544433333443444 
00003369 

00003390  1664  222165654633446335445444663544 
00003390 

00001194  1465  211334356655554755555666566677 
00001194 

00003392  1354  221144444421444335544343444664 
00003392 

00001203  2354  22114441664444464545545555 
00001203 

00001313  2543  115144444444444444423444224444 

00001313 

00003457  1455  221144444541444444444444444444 

00003457 

00001384  1465  2231564  6644444645444444444444 

00001384 

00003467  1475  221144444532454444444444444444 

00003467 

00001323  1365  221344444411444444444444447777 
00001323 

00008021  1454  211355543334553544455445444 
00008021 

00003294  2133  123244454444444444444444444444 

00003294 

00001279  1535  222133323435442354444445444344 
00001279 

00001310  1373  222353351131345323155333322134 
00001310 

00001263  2334  2141435466535434  5555555555444 
00001263 

00003725  1564  221155554431455445555444556555 
00003725 

00003387  1564  221154346623241122312221122222 
00003387 
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00003302  2744  222144444441444444714114414414 

00003302 

00003426  15.13  221146444431444544434344553434 
00003426 

00001362  1454  224166666654646566555554665666 
00001362 

00003746  2452  122166555522444646544444556666 
00003746 

00001373  2335  223132447611144214432333333333 
00001373 

00001370  1263  223355535552646516454644655545 
00001370 

00003408  1243  113265666667756757666666566467 
00003408 

00003324  1342  124143446444446664546644577444 
00003324 

00003332  2254  21437  66  455566736  7777676  6 
00003332 

00003391  2325  222143534542434424555655554555 
00003391 

00003346  2224  211176765566546757677777777776 
00003346 

00003263  2332  221142433721433345444433544444 
00003263 

00001389  1463  224166555544456656666565665456 
00001389 

00003403  2442  214166676636637337666675776775 
00003403 

00003497  2433  121141444332233444311231121121 
00003497 

00003427  1463  221122344431443323444343444742 
00003427 

00008004  1463  21135133336544223345535523444 
00008004 

00003273  1234  2213653532615  65266553546647  6 
00003273 

00008022  1365  21333356776736566535565566 
00008022 

00008020  1454  22115555543334553544443444 
00008020 

00003716  2333  223231356563216316677771124543 
00003716 

00003411  1353  225165346664444414344444343363 
00003411 

00003383  1465  212177777774647777677567773747 
00003383 

00003446  1564  213366676776776666676676775676 
00003446 

00003736  2334  212133336653542213355544333433 
00003736 

00002041  1233  224173667754747767567677777777 
00002041 

00002047  1354  222  65667751666666655646776642 
00002047 

00003279  1354  221334366634545645545454554553 
00003279 

00003418  2644  221135453431454345555455534465 
00003418 

00001381  1664  213154555544455635555364554744 
00001381 

00001246  1355  221176666574547656577767657775 
00001246 

00003282  2434  221121426464144614464641416726 
00003282 

00001207  2435  213141414414141111444441444444 
00001207 


00003747  2534  1121555S44S5445S35555543  54443 
00003747 

00001265  2344  223164554453556646666666665443 
00001265 

00003730  1453  225153543342243533344534434453 
00003730 

00003723  1364  221354634564636656577747777564 
00003723 

00003410  1361  213365676767777626466656776766 
00003410 

00003491  1462  222333344434354342222232333445 
00003491 

00003295  1365  221365664444445656555555556666 
00003295 

00003283  2435  213122223442142432  11411115513 
00003283 

00003314  1532  222122214421242442323323225322 
00003314 

00003311  1235  222273747741455567574775554453 
00003311 

00003456  1555  211154447753445625566656546655 
00003456 

00003350  2524  212154556666545445566665555454 
00003350 

00003368  2324  221145564431445446544454554734 
00003368 

00002049  1234  125177777777777777777777777777 
00002049 

00002043  1455  22217777777  477777777777776777 
00002043 

00002048  1255  225163454374445524363546455666 
00002048 

00003327  2455  225177777777767777777777777777 
00003327 

00002040  1334  2251665666445465  6576557776676 
00002040 

00003488  2325  22216666  54555566  66  47447 
00003488 

00003447  2153  123155556754  6655666555666666 

00003447 

00003347  2545  111166666555666666665566665565 
00003347 

00001305  1341  221275766654647757477657776576 
00001305 

00003729  1465  223311166675361111477771114717 
00003729 

00001368  2255  223121221251113131122414112222 
00001368 

00001214  2264  214152663255645535566555663622 
00001214 

00001204  1564  114152356644545315522332543544 
00001204 

00001283  1324  112166767546676566656465666666 
00001283 

00003715  1244  222221151351121213555551124546 
00003715 

00003293  1552  212141242335414523455543554653 
00003293 

00003444  1574  213365554633544535444543554454 
00003444 

00001196  2555  2211635455334  4534454544544367 
00001196 

00003318  1364  221343543521132232332332122323 
00003318 

00003466  1465  221354523321465515445453556646 
00003466 


95 


000013^2  1365  221344554344445535555555554464 
00001372 

00003337  1235  121255555554555455555555555555 
00003337 

00001225  1554  222144626652446655455555553655 
00001225 

00003450  1654  2111  5456656456765  6656566  56 
00003450 

00003423  1543  121143344521443314424242442555 
00003423 

00003304  1454  212177774477777757777777777777 
00003304 

00003288  1255  222366333531545535535555354766 
00003288 

00001308  2324  221163664637767776676666775344 
00001308 

00001321  1334  222356444531253335366455445656 
00001321 

00001290  1234  211236363656566665666665565555 
00001290 

00003430  2333  211136364356343765544451433545 
00003430 

00001347  1434  214164667755666657766666663466 
00001347 

00001367  1364  11236564364666663666665655 
00001367 

00001278  2234  213152523424443332455444555444 
00001278 

00003493  1664  2231433545215  53  3433  32444443 
0000341»3 

00001306  1463  221356664321466646655252564245 
00001306 

00001373  1243  22125664664  464655555566556666 
00001373 

00001243  1434  211332357114532123212231232476 
00001243- 

00003312  2325  2112636476324337466556666645 
00003312 

00001313  1465  125353447736543625564647657774 
00001315 

00003439  2263  221154656541435414455242545455 
00003439 

00001271  2635  211177777777757777777777777777 
00001271 

00001197  2462  221163656643445556666566556454 
00001197 

00003286  1265  225166667764657666655766676676 
00003286 

00003328  1455  221111114344141111111111111111 
00003328 

00003300  1443  221256655454555557577677776666 
00003300 

00003374  1334  222266634651446656555666666666 
00003374 

00003303  2435  221155557744466465555565556665 
00003303 

00003280  1253  211364665656545667667667665564 
00003280 

00003285  2735  2211434345433445  3444555545554 
00003285 

00003262  1534  224164664553456656577776775776 
00003262 

00003460  2254  222115454551465665555665545545 
00003460 

00001224  1255  22114234667  144113432655444114 
00001224 
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00001330  1465  211174665565545647545665666566 
00001330 

00002044  2425  223122234454243522222444447744 
00002044 

00003361  1255  2141324  4464444513477555346666 
00003361 

00001273  1464  122366666643655666455535554564 
00001273 

00001328  1454  224161443242341115435761325435 
00001328 

00003477  2234  221175767772577647777777467777 
00003477 

00003331  2434  221144544431245747466667445447 
00003331 

00001331  1365  214353536653546545444444554544 
00001331 

00001387  1364  221335555344556565553534555355 
00001387 

00003740  2233  121175767734447746677777777777 
00003740 

00003727  1364  2213214465213433234322423335 
00003727 

00001233  2433  222135436623443223322332332222 
00001233 

00001264  2635  223152534431444545555554554565 
00001264 

00001388  2333  221156646644445655333555554643 
00001388 

00003380  1345  222155657744667756544576774576 
00003380 

00003397  1535  213176677664766777656676775666 
00003397 

00003313  1553  123111411111141111111111111111 
00003313 

00003303  1644  211144446644444445545444554444 
00003305 

00003481  2354  222177774454757767777777777477 
00003481 

00007998  2643  121155577722445244444444447777 
00007998 

00003424  1663  221133351231244333311111122334 
00003424 

00003476  1565  221343343524443544555443344555 
00003476 

00003421  1265  221355556651575545566555555534 
00003421 

00001274  1465  222366664454566656566666666474 
00001274 

00003405  1434  123244455534455434355554553777 
00003405 

00001200  1465  225164555535666635555555664333 
00001200 

00001262  2454  113177774445474555533443454444 
00001262 

00003274  1455  222154647661456665666654665456 
00003274 

00003307  335  221154343341446535455544446564 

00003307 

00001261  1464  223356714655345777666766545244 
00001261 
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000032C9  1334  223177777774777777777777777777 
00003269 

00003461  1345  223264S77777S3S725576667S57777 
00003461 

00002030  2333  223166667774646636666666665665 
00002050 

00003440  1354  223163666733336643666367666445 
00003440 

00003343  1234  223166676776756757777676776367 
00003343 

00003367  2435  223177777777777777777777777777 
00003367 

00003463  2463  223367677764677777677677776465 
00003463 

00003270  1363  223173676674667756666666663566 
00003270 
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Appendix  D:  Correlation  Coefficients 


Correlation  Analysis 

Cronbach  Coefficient  Alpha 

for  RAN  variables  0.962413 

for  STAMOAROIZED  variables:  0.962832 


Raw  Variables 


Std.  Variables 


Deleted 

variable 

Correlation 
with  Total 

Alpha 

Correlation 
with  Total 

Alpha 

QIO 

0.726432 

0.960673 

0.728011 

0.961099 

Qll 

0.633258 

0.961525 

0.635552 

0.961891 

Q12 

0.733520 

0.960654 

0.734384 

0.961044 

Q13 

0.655216 

0.961271 

0.655547 

0.961721 

Q14 

0.528811 

0.962275 

0.531884 

0.962771 

QIS 

0.531298 

0.962273 

0.534320 

0.962750 

Q16 

0.612784 

0.961643 

0.609208 

0.962116 

Q17 

0.478262 

0.963243 

0.478898 

0.963217 

Q18 

0.719111 

0.960725 

0.719778 

0.961170 

019 

0.433458 

0.962827 

0.433474 

0.963597 

Q20 

0.789469 

0.960135 

0.790426 

0.960559 

021 

0.787121 

0.960083 

0.784869 

0.960608 

022 

0.592484 

0.962014 

0.593166 

0.962252 

023 

0.827542 

0.959810 

0.828518 

0.960229 

024 

0.760724 

0.960518 

0.762147 

0.960804 

025 

0.761387 

0.960332 

0.761922 

0.960806 

026 

0.773632 

0.960258 

0.772983 

0.960711 

027 

0.763309 

0.960338 

0.760742 

0.960816 

028 

0.824436 

0.959914 

0.825974 

0.960251 

029 

0.810359 

0.959841 

0.807557 

0.960411 

030 

0.815357 

0.959845 

0.814650 

0.960349 

031 

0.829597 

0.959747 

0.828536 

0.960228 

032 

0.732901 

0.960597 

0.731041 

0.961073 

033 

0.549436 

0.962166 

0.549851 

0.962619 

034 

0.718442 

0.960722 

0.716270 

0.961200 

035 

0.638328 

0.961418 

0.638563 

0.961866 
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Comlation  Analysis 


Pssrson  Corrslstion  Coafficisnts  / 

Prob  >  |R| 

under  Ho: 

Rbo^O  / 

N  -  223 

010 

Qll 

Q12 

Q13 

Q14 

Q15 

Q16 

QIO 

1.00000 

0.0 

0.58958 

0.0001 

0.74453 

0.0001 

0.55194 

0.0001 

0.39887 

0.0001 

0.40610 

0.0001 

0.37492 

0.0001 

Qll 

0.58958 

0.0001 

1.00000 

0.0 

0.56297 

0.0001 

0.50189 

0.0001 

0.36133 

0.0001 

0.44118 

0.0001 

0.28135 

0.0001 

Q12 

0.74453 

0.0001 

0.56297 

0.0001 

1.00000 

0.0 

0.53096 

0.0001 

0.43143 

0.0001 

0.37827 

0.0001 

0.36917 

0.0001 

Q13 

0.55194 

0.0001 

0.50189 

0.0001 

0.53096 

0.0001 

1.00000- 

0.0 

0.32747 

0.0001 

0.27253 

0.0001 

0.48202 

0.0001 

Q14 

0.39887 

0.0001 

0.36133 

0.0001 

0.43143 

0.0001 

0.32747 

0.0001 

1.00000 

0.0 

0.74867 

0.0001 

0.37354 

0.0001 

Q15 

0.40610 

0.0001 

0.44118 

0.0001 

0.37827 

0.0001 

0.27253 

0.0001 

0.74867 

0.0001 

1.00000 

0.0 

0.32345 

0.0001 

Q16 

0.37492 

0.0001 

0.28135 

0.0001 

0.36917 

0.0001 

0.48202 

0.0001 

0.37354 

0.0001 

0.32345 

0.0001 

1.00000 

0.0 

Q17 

0.35593 

0.0001 

0.27907 

0.0001 

0.31078 

0.0001 

0.39610 

0.0001 

0.35137 

0.0001 

0.30481 

0.0001 

0.55475 

0.0001 

Q18 

0.57235 

0.0001 

0.51645 

0.0001 

0.51450 

0.0001 

0.58777 

0.0001 

0.40589 

0.0001 

0.38004 

0.0001 

0.42752 

0.0001 

019 

0.32585 

0.0001 

0.42203 

0.0001 

0.36188 

0.0001 

0.34165 

0.0001 

0.32483 

0.0001 

0.35962 

0.0001 

0.23284 

0.0005 

020 

0.63768 

0.0001 

0.55512 

0.0001 

0.68766 

0.0001 

0.59694 

0.0001 

0.41374 

0.0001 

0.42905 

0.0001 

0.45583 

0.0001 

021 

0.58522 

0.0001 

0.54364 

0.0001 

0.61385 

0.0001 

0.48273 

0.0001 

0.32240 

0.0001 

0.37402 

0.0001 

0.53968 

0.0001 

022 

0.43681 

0.0001 

0.53650 

0.0001 

0.55642 

0.0001 

0.45246 

0.0001 

0.28686 

0.0001 

0.29876 

0.0001 

0.32330 

0.0001 

023 

0.69373 

0.0001 

0.59733 

0.0001 

0.71822 

0.0001 

0.60780 

0.0001 

0.42544 

0.0001 

0.40809 

0.0001 

0.44430 

0.0001 

024 

0.60820 

0.0001 

0.50076 

0.0001 

0.60074 

0.0001 

0.52041 

0.0001 

0.40888 

0.0001 

0.41968 

0.0001 

0.43843 

0.0001 

025 

0.53729 

0.0001 

0.36841 

0.0001 

0.56320 

0.0001 

0.44684 

0.0001 

0.33953 

0.0001 

0.36999 

0.0001 

0.57999 

0.0001 

026 

0.55002 

0.0001 

0.42004 

0.0001 

0.55777 

0.0001 

0.46692 

0.0001 

0.33283 

0.0001 

0.35673 

0.0001 

0.55209 

0.0001 
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010 

Qll 

Q12 

Q13 

Q14 

Q15 

Q27 

0.50876 

0.0001 

0.33303 

0.0001 

0.49822 

0.0001 

0.42333 

0.0001 

0.39744 

0.0001 

0.41586 

0.0001 

Q28 

0.57880 

0.0001 

0.47362 

0.0001 

0.59868 

0.0001 

0.53184 

0.0001 

0.47197 

0.0001 

0.48782 

0.0001 

Q29 

0.55881 

0.0001 

0.53044 

0.0001 

0.58571 

0.0001 

0.45715 

0.0001 

0.38578 

0.0001 

0.43316 

0.0001 

030 

0.62170 

0.0001 

0.55703 

0.0001 

0.63719 

0.0001 

0.53446 

0.0001 

0.35441 

0.0001 

0.38337 

0.0001 

031 

0.62147 

0.0001 

0.56213 

0.0001 

0.64504 

0.0001 

0.58481 

0.0001 

0.36575 

0.0001 

0.40509 

0.0001 

032 

0.49696 

0.0001 

0.41746 

0.0001 

0.47391 

0.0001 

0.40628 

0.0001 

0.38926 

0.0001 

0.35073 

0.0001 

033 

0.28880 

0.0001 

0.30424 

0.0001 

0.30277 

0.0001 

0.39613 

0.0001 

0.34956 

0.0001 

0.33479 

0.0001 

034 

0.59568 

0.0001 

0.45450 

0.0001 

0.53078 

0.0001 

0.49225 

0.0001 

0.37171 

0.0001 

0.33074 

0.0001 

035 

0.44858 

0.0001 

0.37279 

0.0001 

0.42398 

0.0001 

0.43881 

0.0001 

0.32430 

0.0001 

0.29373 

0.0001 

* 


■» 


-  223 
Q16 

0.68189 

0.0001 

0.52220 

0.0001 

0.58124 

0.0001 

0.41013 

0.0001 

0.40679 

0.0001 

0.50286 

0.0001 

0.35434 

0.0001 

0.40605 

0.0001 

0.40372 

0.0001 
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on 

Q18 

Q19 

Q20 

Q21 

Q22 

Q23 

QIO 

0.35593 

0.0001 

0.57235 

0.0001 

0.32585 

0.0001 

0.63768 

0.0001 

0.58522 

0.0001 

0.43681 

0.0001 

0.69373 

0.0001 

Qll 

0.27907 

0.0001 

0.51645 

0.0001 

0.42203 

0.0001 

0.55512 

0.0001 

0.54364 

0.0001 

0.53650 

0.0001 

0.59733 

0.0001 

Q12 

0.31078 

0.0001 

0.51450 

0.0001 

0.36188 

0.0001 

0.68766 

0.0001 

0.61385 

0.0001 

0.55642 

0.0001 

0.71822 

0.0001 

Q13 

0.39610 

0.0001 

0.58777 

0.0001 

0.34165 

0.0001 

0.59694 

0.0001 

0.48273 

0.0001 

0.45246 

0.0001 

0.60780 

0.0001 

Q14 

0.35137 

0.0001 

0.40589 

0.0001 

0.32483 

0.0001 

0.41374 

0.0001 

0.32240 

0.0001 

0.28686 

0.0001 

0.42544 

0.0001 

Q13 

0.30481 

0.0001 

0.38004 

0.0001 

0.35962 

0.0001 

0.42905 

0.0001 

0.37402 

0.0001 

0.29876 

0.0001 

0.40809 

0.0001 

Q16 

0.55475 

0.0001 

0.42752 

0.0001 

0.23284 

0.0005 

0.45583 

0.0001 

0.53968 

0.0001 

0.32330 

0.0001 

0.44430 

0.0001 

Q17 

1.00000 

0.0 

0.42848 

0.0001 

0.26478 

0.0001 

0.30488 

0.0001 

0.35414 

0.0001 

0.29072 

0.0001 

0.28987 

0.0001 

Q18 

0.42848 

0.0001 

1.00000 

0.0 

0.37549 

0.0001 

0.63169 

0.0001 

0.53822 

0.0001 

0.47113 

0.0001 

0.66026 

0.0001 

Q19 

0.26478 

0.0001 

0.37549 

0.0001 

1.00000 

0.0 

0.37391 

0.0001 

0.34848 

0.0001 

0.34427 

0.0001 

0.32040 

0.0001 

Q20 

0.30488 

0.0001 

0.63169 

0.0001 

0.37391 

0.0001 

1.00000 

0.0 

0.68927 

0.0001 

0.57539 

0.0001 

0.78990 
.  0.0001 

021 

0.35414 

0.0001 

0.53822 

0.0001 

0.34848 

0.0001 

0.68927 

0.0001 

1.00000 

0.0 

0.59320 

0.0001 

0.71292 

0.0001 

022 

0.29072 

0.0001 

0.47113 

0.0001 

0.34427 

0.0001 

0.57539 

0.0001 

0.59320 

0.0001 

1.00000 

0.0 

0.61756 

0.0001 

023 

0.28987 

0.0001 

0.66026 

0.0001 

0.32040 

0.0001 

0.78990 

0.0001 

0.71292 

0.0001 

0.61756 

0.0001 

1.00000 

0.0 

024 

0.32365 

0.0001 

0.62719 

0.0001 

0.34454 

0.0001 

0.63658 

0.0001 

0.62393 

0.0001 

0.51483 

0.0001 

0.72735 

0.0001 

025 

0.38066 

0.0001 

0.48757 

0.0001 

0.28861 

0.0001 

0.56493 

0.0001 

0.60988 

0.0001 

0.34713 

0.0001 

0.61061 

0.0001 

026 

0.37883 

0.0001 

0.52785 

0.0001 

0.29237 

0.0001 

0.54684 

0.0001 

0.62709 

0.0001 

0.38102 

0.0001 

0.61711 

0.0001 
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017 

Q18 

Q19 

Q20 

Q21 

Q22 

Q23 

Q27 

0.45198 

0.0001 

0.46667 

0.0001 

0.23696 

0.0004 

0.53580 

0.0001 

0.61540 

0.0001 

0.38059 

0.0001 

0.62707 

0.0001 

Q28 

0.38779 

0.0001 

0.57632 

0.0001 

0.37486 

0.0001 

0.60992 

0.0001 

0.64258 

0.0001 

0.44054 

0.0001 

0.70535 

0.0001 

029 

0.38738 

0.0001 

0.57983 

0.0001 

0.34053 

0.0001 

0.66195 

0.0001 

0.73964 

0.0001 

0.54035 

0.0001 

0.67790 

0.0001 

030 

0.36092 

0.0001 

0.74657 

0.0001 

0.32614 

0.0001 

0.72991 

0.0001 

0.69246 

0.0001 

0.52044 

0.0001 

0.72895 

0.0001 

031 

0.38432 

0.0001 

0.72265 

0.0001 

0.35526 

0.0001 

0.76218 

0.0001 

0.68790 

0.0001 

0.56212 

0.0001 

0.74907 

0.0001 

032 

0.33571 

0.0001 

0.45334 

0.0001 

0.31298 

0.0001 

0.52770 

0.0001 

0.64503 

0.0001 

0.40561 

0.0001 

0.56304 

0.0001 

033 

0.20822 

0.0018 

0.33876 

0.0001 

0.18291 

0.0062 

0.41604 

0.0001 

0.42755 

0.0001 

0.15754 

0.0186 

0.39461 

0.0001 

034 

0.37326 

0.0001 

0.50051 

0.0001 

0.22802 

0.0006 

0.53430 

0.0001 

0.56510 

0.0001 

0.39142 

0.0001 

0.58636 

0.0001 

035 

0.26694 

0.0001 

0.40918 

0.0001 

0.23770 

0.0003 

0.49613 

0.0001 

0.49361 

0.0001 

0.31645 

0.0001 

0.54249 

0.0001 

f 
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024 

025 

026 

Q27 

Q20 

Q29 

QIO 

0.60820 

0.0001 

0.53729 

0.0001 

0.55002 

0.0001 

0.50876 

0.0001 

0.57880 

0.0001 

0.55881 

0.0001 

Qll 

0.50076 

0.0001 

0.36841 

0.0001 

0.42004 

0.0001 

0.33303 

0.0001 

0.47362 

0.0001 

0.53044 

0.0001 

Q12 

0.60074 

0.0001 

0.56320 

0.0001 

0.55777 

0.0001 

0.49822 

0.0001 

0.59868 

0.0001 

0.58571 

0.0001 

Q13 

0.52041 

0.0001 

0.44684 

0.0001 

0.46692 

0.0001 

0.42333 

0.0001 

0.53184 

0.0001 

0.45715 

0.0001 

Q14 

0.40888 

0.0001 

0.33953 

0.0001 

0.33283 

0.0001 

0.39744 

0.0001 

0.47197 

0.0001 

0.38578 

0.0001 

Q15 

0.41968 

0.0001 

0.36999 

0.0001 

0.35673 

0.0001 

0.41586 

0.0001 

0.48782 

0.0001 

0.43316 

0.0001 

Q16 

0.43843 

0.0001 

0.57999 

0.0001 

0.55209 

0.0001 

0.68189 

0.0001 

0.52220 

0.0001 

0.58124 

0.0001 

Q17 

0.32365 

0.0001 

0.38066 

0.0001 

0.37883 

0.0001 

0.45198 

0.0001 

0.38779 

0.0001 

0.38738 

0.0001 

Q18 

0.62719 

0.0001 

0.48757 

0.0001 

0.52785 

0.0001 

0.46667 

0.0001 

0.57632 

0.0001 

0.57983 

0.0001 

Q19 

0.34454 

0.0001 

0.28861 

0.0001 

0.29237 

0.0001 

0.23696 

0.0004 

0.37486 

0.0001 

0.34053 

0.0001 

020 

0.63658 

0.0001 

0.56493 

0.0001 

0.54684 

0.0001 

0.53580 

0.0001 

0.60992 

0.0001 

0.66195 

0.0001 

021 

0.62393 

0.0001 

0.60988 

0.0001 

0.62709 

0.0001 

0.61540 

0.0001 

0.64258 

0.0001 

0.73964 

0.0001 

Q22 

0.51483 

0.0001 

0.34713 

0.0001 

0.38102 

0.0001 

0.38059 

0.0001 

0.44054 

0.0001 

0.54035 

0.0001 

023 

0.72735 

0.0001 

0.61061 

0.0001 

0.61711 

0.0001 

0.62707 

0.0001 

0.70535 

0.0001 

0.67790 

0.0001 

024 

1.00000 

0.0 

0.62300 

0.0001 

0.66321 

0.0001 

0.57009 

0.0001 

0.69141 

0.0001 

0.57888 

0.0001 

025 

0.62300 

0.0001 

1.00000 

0.0 

0.88812 

0.0001 

0.80324 

0.0001 

0.76298 

0.0001 

0.68661 

0.0001 

026 

0.66321 

0.0001 

0.88812 

0.0001 

1.00000 

0.0 

0.75206 

0.0001 

0.77244 

0.0001 

0.68018 

0.0001 

-  223 

Q30  « 

0.62170 

0.0001 

0.55703 

0.0001 

0.63719 

0.0001 

0.53446 

0.0001 

0.35441 

0.0001 

0.38337 

0.0001 

0.41013 

0.0001 

0.36092 

0.0001 

0.74657 

0.0001 

0.32614 

0.0001 

0.72991 

0.0001 

0.69246 

0.0001 

0.52044 

0.0001 

0.72895 

0.0001 

0.66703 

0.0001 

0.58612 

0.0001 

0.62162 

0.0001 
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Q24 

Q25 

026 

Q27 

0.57009 

0.0001 

0.80324 

0.0001 

0.75206 

0.0001 

Q28 

0.69141 

0.0001 

0.76298 

0.0001 

0.77244 

0.0001 

Q29 

0.57888 

0.0001 

0.68661 

0.0001 

0.68018 

0.0001 

Q30 

0.66703 

0.0001 

0.38612 

0.0001 

0.62162 

0.0001. 

Q31 

0.66971 

0.0001 

0.60035 

0.0001 

0.60728 

0.0001 

Q32 

0.56637 

0.0001 

0.62362 

0.0001 

0.64249 

0.0001 

Q33 

0.38747 

0.0001 

0.49203 

0.0001 

0.48739 

0.0001 

Q34 

0.47712 

0.0001 

0.56604 

0.0001 

0.57203 

0.0001 

Q35 

0.48782 

0.0001 

0.34666 

0.0001 

0.36920 

0.0001 

Q27 

Q28 

Q29 

Q30 

1.00000 

0.0 

0.77984 

0.0001 

0.69670 

0.0001 

0.58333 

0.0001 

0.77984 

0.0001 

1.00000 

0.0 

0.65875 

0.0001 

0.67138 

0.0001 

0.69670 

0.0001 

0.65875 

0.0001 

1.00000 

0.0 

0.71137 

0.0001 

0.58353 

0.0001 

0.67138 

0.0001 

0.71157 

0.0001 

1.00000 

0.0 

0.38938 

0.0001 

0.67801 

0.0001 

0.71420 

0.0001 

0.89758 

0.0001 

0.68342 

0.0001 

0.61002 

0.0001 

0.68011 

0.0001 

0.62277 

0.0001 

0.50494 

0.0001 

0.33264 

0.0001 

0.46919 

0.0001 

0.30710 

0.0001 

0.59034 

0.0001 

0.61893 

0.0001 

0.63333 

0.0001 

0.60121 

0.0001 

0.32657 

0.0001 

0.39332 

0.0001 

0.48766 

0.0001 

0.50472 

0.0001 
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t 


031 

Q32 

Q33 

Q34 

Q33 

QIO 

0.62147 

0.0001 

0.49696 

0.0001 

0.28880 

0.0001 

0.39368 

0.0001 

0.44838 

0.0001 

Qll 

0.36213 

0.0001 

0.41746 

0.0001 

0.30424 

0.0001 

0.43430 

0.0001 

0.37279 

0.0001 

Q12 

0.64304 

0.0001 

0.47391 

0.0001 

0.30277 

0.0001 

0.33078 

0.0001 

0.42398 

0.0001 

Q13 

0.38481 

0.0001 

0.40628 

0.0001 

0.39613 

0.0001 

0.49223 

0.0001 

0.43881 

0.0001 

Q14 

0.3637S 

0.0001 

0.38926 

0.0001 

0.34936 

0.0001 

0.37171 

0.0001 

0.32430 

0.0001 

QIS 

0.40309 

0.0001 

0.33073 

0.0001 

0.33479 

0.0001 

0.33074 

0.0001 

0.29373 

0.0001 

Q16 

0.40679 

0.0001 

0.30286 

0.0001 

0.33434 

0.0001 

0.40603 

0.0001 

0.40372 

0.0001 

Q17 

0.38432 

0.0001 

0.33371 

0.0001 

0.20822 

0.0018 

0.37326 

0.0001 

0.26694 

0.0001 

Qia 

0.72263 

0.0001 

0.45334 

0.0001 

0.33876 

0.0001 

0.30031 

0.0001 

0.40918 

0.0001 

Q19 

0.33326 

0.0001 

0.31298 

0.0001 

0.18291 

0.0062 

0.22802 

0.0006 

0.23770 

0.0003 

Q20 

0.76218 

0.0001 

0.32770 

0.0001 

0.41604 
'  0.0001 

0.33430 

0.0001 

0.49613 

0.0001 

021 

0.68790 

0.0001 

0.64303 

0.0001 

0.42733 

0.0001 

0.36310 

0.0001 

0.49361 

0.0001 

022 

0.36212 

0.0001 

0.40361 

0.0001 

0.13734 

0.0186 

0.39142 

0.0001 

0.31643 

0.0001 

023 

0.74907 

0.0001 

0.36304 

0.0001 

0.39461 

0.0001 

0.38636 

0.0001 

0.54249 

0.0001 

024 

0.66971 

0.0001 

0.36637 

0.0001 

0.38747 

0.0001 

0.47712 

0.0001 

0.48782 

0.0001 

023 

0.60033 

0.0001 

0.62362 

0.0001 

0.49203 

0.0001 

0.36604 

0.0001 

0.54666 

0.0001 

026 

0.60728 

0.0001 

0.64249 

0.0001 

0.48739 

0.0001 

0.37203 

0.0001 

0.56920 

0.0001 
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Q31 

Q32 

Q33 

Q34 

Q39 

Q27 

0.98938 

0.0001 

0.68342 

0.0001 

0.90494 

0.0001 

0.99034 

0.0001 

0.92697 

0.0001 

Q28 

0.87801 

0.0001 

0.61002 

0.0001 

0.93264 

0.0001 

0.61899 

0.0001 

0.99332 

0.0001 

029 

0.71420 

0.0001 

0.68011 

0.0001 

0.46919 

0.0001 

0.63333 

0.0001 

0.48766 

0.0001 

030 

0.89798 

0.0001 

0.62277 

0.0001 

0.90710 

0.0001 

0.60121 

0.0001 

0.90472 

0.0001 

Q31 

1.00000 

0.0 

0.97497 

0.0001 

0.48278 

0.0001 

0.69749 

0.0001 

0.93013 

0.0001 

032 

0.97497 

0.0001 

1.00000 

0.0 

0.61149 

0.0001 

0.64822 

0.0001 

0.99699 

0.0001 

033 

0.48278 

0.0001 

0.61149 

0.0001 

1.00000 

0.0 

0.49244 

0.0001 

0.99624 

0.0001 

Q34 

0.69749 

0.0001 

0.64822 

0.0001 

0.49244 

0.0001 

1.00000 

0.0 

0.66774 

0.0001 

Q3S 

0.93013 

0.0001 

0.99699 

0.0001 

0.99624 

0.0001 

0.66774 

0.0001 

1.00000 

0.0 

f 
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